From 6e1e49ad72c3430a782ec1e878ca445d132492ff Mon Sep 17 00:00:00 2001 From: Willem Pienaar Date: Fri, 14 Dec 2018 07:53:19 +0530 Subject: [PATCH] Added content to README.md and added components.md and concepts.md to documentation --- README.md | 30 +++++++++++++++++++++++++++++- docs/architecture.png | Bin 0 -> 24680 bytes docs/components.md | 35 +++++++++++++++++++++++++++++++++++ docs/concepts.md | 31 +++++++++++++++++++++++++++++++ 4 files changed, 95 insertions(+), 1 deletion(-) create mode 100644 docs/architecture.png create mode 100644 docs/components.md create mode 100644 docs/concepts.md diff --git a/README.md b/README.md index e57dab254d3..f2889f129c7 100644 --- a/README.md +++ b/README.md @@ -11,11 +11,39 @@ It aims to: * Enable discovery and documentation of features * Provide an overview of the general health of features in the system +## High Level Architecture + +![Feast Architecture](docs/architecture.png) + +The Feast platform is broken down into the following functional areas: + +* __Create__ features based on defined format and programming model +* __Ingest__ features via streaming input, import from files or BigQuery tables, and write to an appropriate data store +* __Store__ feature data for both serving and training purposes based on feature access patterns +* __Access__ features for training and serving +* __Discover__ information about entities and features stored and served by Feast + +## Motivation + +__Access to features in serving__: Machine learning models typically require access to features created in both batch pipelines, and real time streams. Feast provides a means for accessing these features in a serving environment, at low latency and high load. + +__Consistency between training and serving__: In many machine learning systems there exists a disconnect between features that are created in batch pipelines for the training of a model, and ones that are created from streams for the serving of real-time features. By centralizing the ingestion of features, Feast provides a consistent view of both batch and real-time features, in both training and serving. + +__Infrastructure management__: Feast abstracts away much of the engineering overhead associated with managing data infrastructure. It handles the ingestion, storage, and serving of large amount of feature data in a scalable way. The system configures data models based on your registered feature specifications, and ensures that you always have a consistent view of features in both your historical and real-time data stores. + +__Feature standardisation__: Feast presents a centralized platform on which teams can register features in a standardized way using specifications. This provides structure to the way features are defined and allows teams to reference features in discussions with a singly understood link. + +__Discovery__: Feast allows users to easily explore and discover features and their associated information. This allows for a deeper understanding of features and theirs specifications, more feature reuse between teams and projects, and faster experimentation. Each new ML project can leverage features that have been created by prior teams, which compounds an organization's ability to discover new insights. + +## More Information + +* [Components](docs/components.md) +* [Concepts](docs/concepts.md) + ## Notice Feast is still under active development. Your feedback and contributions are important to us. - ## Source Code Headers Every file containing source code must include copyright and license diff --git a/docs/architecture.png b/docs/architecture.png new file mode 100644 index 0000000000000000000000000000000000000000..bc655b60f327b7d7d508885666064e0c713522b5 GIT binary patch literal 24680 zcmc$`WmH^Uw>3xv2@o_`u;A|Q8r%ZG-66QU2Y08?5G(}u6a^ISg%$4Z?oL;p_qq4G zJx2HFAN`H){ijZ?QRi%5Ywo$`sYn$i>G#MVk>TLr-pk5Js=~p&v4DenMUR94v-pom zw8K8$x`@kuMS>MyB(n(EcLLYXTCQpi7Ooz~&gO8I_6~ODOfII*=H~V;Rt~NwZ@|KE zaG&60CB?paW*;sEeEn*fd-C8B^!fANS611%fqZ@@4W7shNrt@;SLx2(iq#y1;*xP| z^vrTGi)JzxUBP5M+a2b9{<0yT-Y~)wEFWs;ZK7(LpUE7-HXpNry7|0!$= zG2^cy{wa%lDH)OfT;VH30{&y``2XItO+egkhUybv7Z4ysTylRFIy{?{UB09uTIrHM zo!i+nvMPSM_H`5upUc_p0bp{kVLb9UEplbdI? zbZ2TNZhh|xlbt5vc9))8@PU!x=-bieJ;KLEwXe|k16gw*j_}V?h?LIZ8BY`zA$@Z| zKfKr+zqjkyvPv)4|^zeS%*$dd2KRfESTNo1w%^; z5pQd}#BC)v(Mt2cIP9fFuzTJQ)Am^_$z#eoVK%};+s#dF)~Z`!pWp5^E}A$jA|U+J z@rha4_FXu0C48&W{2Gz(d|n)>Q|gDMzb#JCbIF6q_-}Y7Y+M7$$<<8oN5?mA1A9i> z6s9V34u8*g-J}DIesyR8z7_r4yER_N2K~ zzPBXNAyW6zImNo*ZnC?V(3@uRTH~= zG)79F1_1KL<2p&}vG-g34e!_amK7u53;lhVOgX%8D~G9lv_<+PULm{=aXoqD_^m{f z)I2YBat39jFYj&bRX7>_UFv_tyVQnp7CEH#WQ#dk5uop>4W&06{b@ChwxU{mpB{Z5?*h~S3I#%5pk$G%Z%fnLq;E1yHiX&PLW%hYJbGJVK z760!W6~Fo{vJ3L;GORtjeKXoCc!Q<3C|-o!U&}+!8zlrtdgI4z5j#^tUrB`&U$kXp z^m_M$nFMzZO-JkObw*yhs$E$Up0=FQFKAkOFWAvxnBYg4l9eG3Y3-sIH@5cVS~B#O zz9Cx{)=^aSLu{2GTgPKEbrZuJ<->bLe7JZ@!M~wd%|9m{-luRp1nO@vEJGXA;~H}c zwl_LcQ^w?X3O>drM%~oo&tYp{c<=Lz|1jSH2@AL6{HT-MnIulHAHkvV<%4Hdn29bR=nL&Rfldv}M`IovY)33*z-gH^#Lz(I?Ls|QJBu>g&>MvMn5RJY#lRHT))j!TE zUX>dENY`8vm|K}ckCdfkLLKw_iF~t+7P}15d2@*d$4KckUKD%Vwb|@ZV`GWQz(Otc zC61t>u+EC(fo3fe$eAt3P5)BjnB#)@n6pYm=!adoVsFz{Oj|UZQ5v@eE?UH z1Sm77KS2p4xGbMG1r*gF@t!rm5=f`P#CyYN&oNy&5(%dtSPtiUM)Z9=*Gk{w^LMP} zEw!J5?6}Ndj<~K*yyEI&nJ&YJ85ukQLINIV&TGiSx4VMNL)9g@1%711L&&rY53fXM z7dpQYqPmgv6f9pc;*fuM&5G^Nw%lluW>>jMYOpYRt^P`9Ux1)fYCGEA|1^YsFjqf} zZ?bPZ%^*Ua8g%;5a@s|DK9Pn;$KYr}E!k8;c+q7;r$tCG9;-Z3{Hag2Wj{`@M{4_8 z=+O1#C!+u+uYAA_+?c$)JSmxN`EpyhzJN!KzOFBV<9B7yPL z9+-;hwmA4i!Jh{xLv?=5t3yau=AzZ?AYB`H>|+a6%@mpu6vD#nCaKl5s;|N%x>yej z0^qa~hJVX+N%{H;T0~58IPmz98STCWMJekw_62po4U}n^*jTS^#-q^M>hu;2Y1G~? z*m!+KnOi1XudbpuvD`;kKDI5asR_kiDzRR?qp)Q;q1+ifSa%FC)9476*1mHr6lQki zmtCh^YF$FDW~|NOOO!5vPE~3mPW^x=nnLlhvr7e06ZR?dYa)-+c3pdGnv)VFYl%Oh z?L<#aI=os(u;TTtdn%F4)oI8m2nUtUsfx; zt{&x0LFe*B&*zCWgqy>HrDqcS>SLJHOZb-7$*PTuO`}|!u~m80{!IH1upgkvOnTSW zP5$f$p1o&hWkr=WwX8@FOE>JnmLs>6zvdDKXAI-zg3OvJ5~?$dM-3L$X^;CEBm{yC~? zO7zV;a#8+7>bhZ!EkQ!5^DSx4*I?HQ8zZ9V+=YRm0cmSnzM!4ScORAWR(B&IEiD}3 z;Z#UjJX9&F^c_2o*%6RosyDUU*X2g#7D?e@?jbU2CYeEE72(J zn1mr3J57k+3;ChGiY()DA>2-|#W5jH^G0Qfx4*VCeO6l?f>MgIS{|O4)7Em9W#p*2T@SY; zjqc+4QbyFqD?P8!;uU;N_qLlV=|hD>?k2wux8Q#(Q!V!qbzac=fx>`3|L_IXM4@CQP#(%laOkz*@M(x2!=Yumm%P3zQOYqMu9 za^{qq$cs@#o^_#KKJX5`*0Am0GX8P$TX{}9CrfUlQc3Cg79Ke(NsD(@zKJDn3gt$% zdNvH;f4}c>-gWGGube~uHlv_a%ePZ}Jt%*}$BQfhl7{!y!>l$*tJRw=_Tjl)8y_r2!D^9!Cqhq-Dto=9WbREr@+1x#UK+vumtlP_p3<7u)vN%_i64 zay~r%N^VQrK;7I-0sw}`GwrXzm#=d_lfkBg5A-SZy{r@O1`JGXeqY|GuXi0=03lfV zvXGD_B*-K&1sz8&t3j!n0JBF6O)oSBA%BWP-tiAGE_Pn_9J*~+DrTM?^k-pG2`BXPhN}Rvj!eWFYNoAx zQI@48@Y)S&UyUndW9-Q%%miEe3bMthm(stq%)>kN!WJ|eu5|@Lo&I&aSo7#WvSql8 zPcA(7InG6R2@;$AI6FAeuwr&6&btB~Rt`9Gx_o-5VRkzoGh$it^YdK=XZ5AioQl0* zQ2OUAsvCB+Ia5W4w_e`7_o~x@P9Q=C*@kk8^)-Mc}`JYhy&w4@GAO|`d zFeo*4sG^&l^x;ic_Vwjy$C%ggLuRx(I)vwbhC+>2c<-QG$2Ul91@ZFw`b_Lo*h5>= zU>MU&qR8{0FMbJHxqfm-?e|yyD^Pid6%4^!UYi#a!WF+xoW5k&rLaoxIdf?OXS5@_cDJ!?#+X~MCN@jnXWxw5gAauK)iJK zxX22EduKSX3QQuOW{d)bK10RRiv8g)5QSRKqLq}c;Uf&2qk><58z_JpDxtyo_o18o zSEq3i5&cIp|4jo}+&SOD9tDXm8LsNw{(M89KGcA4;@XnN-{{xKlfZOhgaejF;R@e>8SI8}9kLu5M z=l)Pos30k6k(xM;KN4s4C}AIi#eoTcy0WOktOY0qQqrGXVqvC#N(dbrYy@G2E#v}Y|Sylk#~4^m$smw;Ct8FUW|NYhZ#F2Hnz)L9=Ws2NMq_*$jkkZ)3R5LVgM+) z{+0k;R#H;(STtE_SV~&@U8p!7HanGNFDpPHLxI}F)D-Br_S(tGsk^s#WN*{9vM)e@ zfPjFF|Lq2&R`)t0p22wtl@>i+!bMdH0nW@6&H26(^hb0QQL9$urx2&?BK@rYt%WaQd0 z=!BaeHS+x7DJCxNBXJnnw$Qnd!Q#DJL01Cbu=flt`mzlhKk6$45tQ?wm0enUeG_ri z>09d-3CiT_j#WN{(UI6^2R%I^6kJADepIp~iMyv{hZySAjL9O$qd6SiRE8UY%h~%Ho|;_;ZtXv0hc8!8v_A& zh}ro!yDKUymnbcy?@ilYFmrE{!J8t5ewcw{C(nQx|MU2kaSl7B>6RL2n>}B$ZGWM@?5D1vLlow`Os1f-P|56cPyfX9>x`G-4wnI!s zH0T?`A~C$e_XxPRQEFM=p7m)zC7f)1;8(`tGml*y1``ZGE1o=goC1C!`naAY? zP2Cu*(Gii6Uz{n};ide6Axr`Hci+PMz(`i_svZ3XQVXOb# zHfex2!U2UmE9nd2^ICg60WV?m?Xrv&zOC|kHwEIZ&q1E&-OD3jqTElfjpe9gg%>2WPbf zwhv3|^5y_HjR8A}0A5O%?*K^DCxu7!OOohy;z8O= z41|47*;Gjal0JkJ1Xy9pz9XHO;ecaYKO1&PI!~oq^w?9Gg{{3OkVkEec(sUuW-RkFhT4i{|k*;^^cFk(~r|-N7hQJ33apa|< z_jGBqO%iI}g0;7-6V!q(Q-9GK|}ouz6etXYsnmr)@t z#x%Z$l!FY>PogF!KmYD;pOaReowhz(TC45F2!t1NT;3ZpdGL9{_!p$p%9}T@T^@ag z;noQ@^Sgm(OWI8!o)s^=?Uh;0tnMu7>R!lWY?HF^-FGVav|&%6bAt-4fP(VKl_T8jKc&?-6^KiF;iCKQvCL037GR}YiYO}zi-Z# zP7@U0?`b~B3Y&Q3Og-rZ19P{dh{~ZV!4dehvt9W~i_tUfi%AlfZyoH?+va`s>_?fx zw{2KxR&ZK{Epb$rKU!}D>%X6DR8sfBD_lA%HZ`ZF+h{wXwA^Ep8xmSvuln4}HqO`P zC6*ZlrL)h>tlM&+HS_YOOP}7BS#jbkEiFwX&7N-7}P)a7^-|F?c&qr#QMZ_T3{rg!4nO>Z12nd`X*M;SP?oiy>f&#|M>|tS?3;U z^k*2EUgA{fYpoKP;~F`G)Fl8nkObJ7)G&U;7dMmGrn#fl8!QtxnSC9bo0GM)q=DeL zf0SRWr+3Noc3R~dYcG0Pcv5t$N8HGW;_wxMjtj5T8m+XwCQ`#*ug@@7BUqqw+xkLi zhyNIUF3Vy~C6kaV(X-jC_h&+X6O2Xh{(#;iQC&;iUxUAN2ufFe6qKuj9Bs3^_dEYj z_4RrA>`9-4Y&R~kclB1vml`huu+5 zdMnIRj^Z6r6*fB$xD+L8E^&0+Ay2scqErpn+cU$FF6mR4x#?Oek738mBJdngz|E4=vi@jAaM$AeD{XzTW&_lch=aLtD{^jHGyk(- zO@qBrqxuTa4}Q$ukG2!naAMLIV+M0`wTNp>&*rRrs-W8Yz(!JY3C+OY$*u2PmE8x#!z}dW9uBs_fHkiQ?{I@y*oc7HZ#u zT@I_;Z+C{I?F{I(y?})!>8H?n!3p_=0)dNF_bt_<_Y*4XKkHLvi2Ip}Ia#7FC%@|T zn9h)pRvfqVk2<5!4J#H^TW8<4C{Et7&((0jE9)P8rzqCg;h$g>a^36X-eQbHod|a0 z)Uze>nPZ1UMj$YevK&8aB=07KeqZVr+x%8;g7~P#-}$4 zTy5-N>w^Y{s(O4Fjy5mC_lJd;V3PTU!qrpe-w5q?nL_Kgyb$asO_oB=Wv!>z13e5D1J7$$Cmj=UKb) z<^>i;_(33?q;pZ`C-#}WVb&q;+$T206~N#W4Iac}=7Kh0GKFrw#Yy-3;iFxV=}cIF z4)G9fd2ZAM23^3@*;Y7g7zNoqUVKK+<6Ir1r zebM;*B`W7C_3E&{e!YU(xCtB)VrN%Rych#+`Mw{GS5UKIi2AlhP#=>o4C3#e?vc=1 z1!!!f&(3X4=d2}X*Iqh_KTj8VcYdFRhm;LIHt9`Hss+YyDzjEU z+y~_aJS_}4Aa>HUb5YTkJFaOvB5V6%J40^$3B!`o9F^Yos`H|Z}jrcjn zc7nZ5_H5E`8QdJaq9zSm5gYuV_$L!3OnOxyz-E06+yS_NYAUf>(fH(~HoqAkC|pL^ ze*~24@->F$g-xh>Pk*++BBrZTRcpy&{!!BW{i6+mH2(r`?FK>lc--W7Hya`zsIdR? z@n&3}vYvY&&z!Ilhr~=qe{oJ)In)jU;TRUDnhIIlD%7o3*p-y^NgGcG^>cT8 zvNnK2xlSGFU*Zmb<8ul*czk`d`IS3)94aTFPh6zEHQRJl@a-uzBop^go>-Dv)021l zxCfdXJ@g*q?SkCn^_8HhlUk-Bhc|2Re`x_EFhqpJP#F5WPp^v-K}xuY3`Fkb)@Y1@ zkmGi&%MHZilmTn-1Cswme&ov;?jjy~rMdC55#20JL)4ksD4vm_SQHMmprKjPnJdF_ zr?2~)HWCqc5hL^ck@OxhxlMP9#(Gp+d!9SQV2sQ43mPdb!RYrT9L)jYZ-&@J#B6!< zYQEHgzB@Z`kA6`9VqDj5-Y}>9a9}VI#ZFN(>pYvGF|zC~0)H(ZPud|o>-chW8L9?z-*{^uAhhy0p9Pti@8M5Mt@fJ!^-b zrIpz&&L~-B*IzIMrOIX`d~%~(;W|FymZc9>DWb90F{N-`sjS*|Z}b-r4{q@!_1ZZ! z@9xSP>(}Gi$!oJIJy?*LLxnRrKvs6Y3Mywm16>di8JNKaJ_`$ErqcC(PSyVU$e?&H zBZ~BTS^xQXuiZU5G6sHjEL8r}yQka7>1YmeM?uBRs_Y3>k-1T`TGPCZb(aR*viH#| zeOk$WAbcR4a6m@p3nn=c+SzTo>8h|m&v)@mbZ0?S`*NogmrUVI0A+`?& zABDuqk1Dk4%Bs7}M?==MWHaBlcc|(ntlW3ZOoeWlm46Y7?l$ZxE;AX8kF;$B5+6CPg=8Gnvi3hMHX ziChSNHwYfT9`7*sBirG<>o!k4DqiT@<9cX&>;ouzP6n4-Ck799WJH7^t6BSdn<=binaf>3W#K6;zwV zx*Z*US=AdINA#pST^qxlIh|{@BOV)PIczR7`WF$gp?kL<>)PTK7~5X9PtaCy@TfgU zx_9C6Z&9DmRuqV0p7WM7aJNgT*c2|AEY!T(PDy``Jy9Q^9aBIN6nlnBLHk{59&OEo z(R>M7Jsreoj6zijy1qw8YRLPgqnBgL^ZmyyOT6xrY8LI92#qa5@yes4k2h;Ye4UKi zoEtrZ#D#Y=c)UEm7kMwcb*yaENcs_!)(+S46FY=k;D+xRq?Y6&rX=wzD#X`N=)BSx zKvr(vJ6BnE^j_lbk2_CI8!b{C`Slc@OjY40=ztSAQ5c zxIOUh1UWBf`(@?H>OO@IE6)9?ll!$dXm&xB1s~GBG~+W*i6q8AAP}blaN27W{`Ss) zxqF_y)XIHgYF^GppG}*KuwZ6pSC}!4(7SR=0vs7|l)G&F>$!t962+md^)ZKKSl@z| zG??`?dijf>U)T4p(^YZomo-9*Hg0{x3%)fz;r^s|qccUeW)=&j*0!tu@u`RO0?BPg z1}Bp4Q(tB_s50`D-iVIHk1hl{JW<0OwORner9WppkMN%wejGf3s4afk>)7i$<`}7G zA_92qQ?NfG`22Oo9~P2lXj7N3u^jV;Sa72Pck$tts2amGH6bB7|EkLA`UC6uUcC`; zKQ?kc*l1JGaJzRwdLCV`E!yckasa6GzzeaM%i314VLr-u4j_;Jw2-vZuozRfYSr#IELNN^ub#on&q z&1)R@|3No#cFyJ0Zn^`gF9in>p@mysPPx(#kFEgDyD)&mB9EgO;9cT1$a9v@({TZc z=8;Rep8yZhb4x`QC(5!E4;83S0|RaS;toc5X%Q)JT#^_$T#E15RBWrv@E)ZKCEI?Z zb#SXb9<+|ZmRs9PVc!2a>Uh_zWRNoqFEp9*hi|iW_I4db2e|MWH<~Il@pBa<7V-4< zRSVJYgU-9*RFwExayv~({;umWGnzWjT36H(NjDx-*NlpSy*1u-%#|Rx07+W2Z^sTi z6W(LNQ(NVl%6dBOQ_%AC3;_#2x~81s2*gIZ|5_nM<-lrv$I3)4*oU1{Ri@1`W9&d* z(whX~nsxxjEbURw>fn`fbTBJfs!;#&Y6663wmhLEqX^eU-oCkDH1Vf(VtGyiK&GjZGSi4Xs}Q7o&Axwaw`X|3rauej$>~) zh=?*+7tU4iYU-R3MOPN@U#wqsb=!4#7XR=gqRIQyCR-m59UjB1lMgR=MU-_@Q`Igu z*sM^l=sz7Bi_hEi1ql2;vFN&pi*g-f_f@CL$#N{QgCors`Qo^EfAs<{eXU{ZtI8Tv zK4lp~07i9jbxrMSn$@9^A1#xgBe(j^J>GWo-bWExFGD3u2=pq^^j;;hIuJ{C+1QvZq*5* zLX4JPPFMIDAGjz$QCb6a=N%6Xj8$?w`N@Cm* z{HJtbjj|;_ywN;t)30l|>*aWxpV0x#Blo!r=IZd_v2Zgy`Ki)MNaD8IY8Ie7_Ga<8 z?1R5go8<=-Z!q67#u3rev)5?lm!DHo6){hQXq3mfrKrcvR8%Je>?}_PY^ej{dwcI| zD_RUP3)=67l=}iGzFG-sa;O)A7p{56kH{U0U`hM%0@nQxiz7SD_zQ z7|S9~sT!2K;7n)7bMb-;%?4qP&i+x^QpvwL-Gu>%AoK%BwPU-Rq+ZXT0IgAN#*FG= zmCBSXjdR*OWz~K+QttxwygOZ7Fwxhv32_}lp|w%THR!$pxH(1g1+kysc-63e45J}I zuA_#ZATna>qRER{KJ}+D8JLB|U7VNRVFsx!yYed94amChga=~BcHys|j1xP|zl`*a z*|fDaY`GbpcdP6k8;urK9a{W|ESV3~@4DR-fBI!x6nAuV985?3PeKb}xGKphhu@7p zGh->F$E1Uk*oDC1%k_(mi2~zHCDqv2j@3$g5V+}pG1i~3>G zM-R_Mw5&EKmpoGBOg{(ENfeGX?Y!eA1#aryrhYBEZa6FQ@?|{zkpWapwq^N(qguhNh zZN%YH)mx5ok?!SXyR~vAZEtMY4}+b#)*}n?+Y){8E{gr~|tOT20 zpwVxOs&Z##n69;77#{;1Tq&FALKdW{EZ8ejQX>10Rx1HHYXTMJ)}w0bFD>K7a7IIm zVxbi_!8HCMv)mfBU0wneJ-5J`y(o_l(IHnjh}sUWL@pTv-;-*s&0HUT!IEr}lBgyf z$1Tns32g4vzVecr+?*(ye|c0`0oKv}4_5IX6XySh^wR8mDhoEY!3_)Sj{qH=Ypl}` zBt|{S+yxRqb30-O$9mw@F6-NBp<9f2!TZ2*V}(JW%g!t~06cEISQ~V4Q@vZM9E-M* z2CMKC;^6Nxv7pS)#*qbK)T&;s&0}E8@-me@pS_ESj?RKcYQwS?e0+Scj0(B)P!g$E z%%(I62R|w>RuFKFC>8$6C1J_F&%TL}xR;3!`#ZRt>&)Zeq?{&FXK-r;`uf`SaMe;;Hv{RcalUFF{daNTh0nUpl^ z#eX&7f9S{S=?fwj6SsRZweU&hd3`M_ZaE3{@k2Y>iSI>&A|hl`U1FAL zfH({;4}OI0*E=HE0W%z$sv>zT6U1Q|gC_QqIVJ|5I2bl2YyTIu{4Y~APA@Tb4K zo$T-U9e?s4eRWN80n2;B8mlJ;d&HuyKQxB5mC?WBHA`-VsMy9lr6G`ReSJk@3 z`g;loFG9I~IYp|aRCy<3+`XbBxLcvA4;DbjqLO5FJ$X%xc6kYFX6E$h%S%1Ek1e+! z<@#3^LaR!@w`f`zc`Cu?o?7wWdUytvUuKYW{hf2|8g}k>4C7b@O5hlg&+Fa)pMPg<}hVez%S6T9IXQUAe=CkY*;7cApw2L^4o2X? zNHLk}%2$R;@DEX{)#e(Gr(PvBeYCJKOb7gvpNH;kaH!1ii9dm36FjRTr|2$jL(FTW z!rkoBKIFRVymFkiyY&;>-TTp(tEe3XU!71piRSPc#v)yLTaCKZ;C;gHHn|eth8Erp z(lK2nKPgZPT=P}G_HKrGS2Op&S^lv5Q#IaIE~g()7B{R|etyiT89xJfKEDtn+>5-c zd-042utl#izYz4L>I}$b;5B zR?Nw-7;0eD6#59-l36!`thQ{F2g`e*EYD@k!h=Gdq+fiwy?acS4iDb$h1zEg8qjYG zl2Xd=LWvJ%n@oj91(;m_#Es)DN@n)F?)Wh-?E>fnza?xl&YfmQ^9a*2n>zHvOax4Q z1~O#Lr?k3iSQt3FjP>O|x;EZ5o!#^D;L=Qd{gd762L8$J@_6Id;I;YWP6#n~acORY z?KQR(zUgjLL*$D-<;BIF@L;xVLZBd<&&|e{On)2#rs!C>a79cr-PEVYdJ&E=}p#Ndv4e4l zH;955JFJ}5+S325)9ula@Z)&0A-;H;^x@DiSOV}@cH`h3m8-Js=o8*X;(U_8PyPq| zzxh3kx|G#qak=CvG&yzWxv#k#Kqh%~w!Z~Ci_4;PM%x;%pw1s)D^XLPtZS|-2!WI@ z%NFjEa}rZwsNLzT!xBdG_Z-?^Vb;&~&*}`s5ecl8w6-K9NeF2jDjoVqY`Tkiqou0U zHlYMy91=48YI%>_kclFMiJ2Kdrn%i@{s^MPVL_nMZ z^WSb2dg(#n%Ao!xD{o#EOYhyrS$9GJPB;bWu-M&Xle}NQ!7~av}l{(VuNZWv)hrSoQHWOoReh3XY34 zQyUJdeBF4DKDMvbC$^9^3AbAE=VCewLQwYFI8zwH)ex<5qjL5QT z=kPDXY4u8MqNSH;;p^XB^tV7Mk$APhBXS1%4Dx;?s?({)8=Hl~b_A9}YeuXQLM7+F zKMVUds9{z)*_U_;|8omF%A7Ozx)puLk*uj{*j188`M-zhbN#F~YBbM7!DNe`V^=I& zho3Y^t38B%)R4q&%Gs|H$>{C>Fapm6!auBkVE#v612C9Y;O1YtoSjZ|@A;Hrss^vS z!brxm^nScYgnRDQgNAz{j%tP*Y8E7n$|{oad2LZ@Ya2_>1?r^BILp!a#hSQen6HZ^jT40k_2KT1kUe2unBm4Et8{=Y#3RZaM! zp#cL2?SQ(SJG+&WzqsahwC~iZU2?(?F9vNZ?N$WE2*@+A%L?=2s6{Y9c`U=02HNx7 z;E@V_&ViKxuUpsa8tw1zSxcHG_!NwG(Qc4t1?d*KeI@|R7`_mCO!ygZI zDCM5)kX;qn!!~jE2>zcXO^NSv_*{B-F)X@6QF<$#%X>s%x5C`6_|wx9k%+&q)X*cu z(r~@ZWAt!lzFeo_o74~}>0ezGVC;NrNW0DVE++Nn8RJpEArg;UNcNDJ6MTZk|Mf2_ z!UP<}Jr2GN-0?s_WbgVnuJ)IoRh5ttg_)Mb#+h0CtF}tiUULOUa)`0M5(((*K8+a1 zgb@83%Gntn4;5JfaM;}$tKAL?QsJDY{}rLvq?jd(*$nVw9|bsHznt5cnr6!WHyR=a zhmOI2BJz7x(#q;O-T%uX?%#H^WqXW$cN3mHL2*fkPr?laS_-=cI9e)LPTu_cj@%k| zVc1*(@`*1JRaHlPUP!crr3c)91E#Y%uj5}+_SD!T2 zf*pYLR+^8Hh>Jf(xA7Bm7(A{%sbvVqPFFTMB$nTt9Z;>s_;=Od2Cxl#Oh^F_Dz#wv zoC+pBLq$hd_I@Cg8-{@_t!kr=_lQ)`jk;0cE?e%uD4GqR07K0yyR4M^i$E8-q|f$A zUYvoD<}nzi|AQgerF{(BSGwFHuy-vEg#HSh<~T^-7_#!m$igIQFhR#+C67EwM9X$_ zVqzk!68qih<=x%IKWdI8e`Q_cWQ(;6YxwpwJ0m$9|MUpsc6b+qlHT+A+G=QIg}Tp4 zfY*oH$Q3Dh`{p4QP0T)NTkjuG81X-F>3=Pc|5v8|^7VqLoNxxAc+KXRIG9ifrWG7y zlpk)U^$_YjpLCsoLS>ale^$?jykzE`P3SC$YMy?~s!vbI4A{+lc=sP+2;LCHgJ-RE zS~$o^^G!oVn~ejdnpx()RCQ42R!_asA2ETVN*O@uQ2J^AxfSsMeVh~_%NgLdyA0@xRy&VBZU0UpmA2m02hYVq7Hy@0ZsOuH*zGbbty22C^fU}jA5W-6 z&ZoDf4>p}iw9k=B_x1`)K(s9yMWFKGgDLRAj4=hIwi~rwY`lYeetzCcK=rN9koBK! zmeHY_R=1gBU*#k!Q|ZFRMn7A$W9pTqkp_1d`D=8-cEZ(|P--r@Bx5S=QaJ2)kiJUG zkIxi9IHEguBikWc#^6PrBTSY6I;VBG*|4D{ z3vj?|E5tbOP5(S58u<-?tFaj&!Bw%6&`Anzljj z(w|g6CKuv~FTZ3Q6vEDaxyDj;iKc7)ee7jT3yGa9{pVOQrY4^cij1Eh8h}(4qn!E} zZrxv%CC*Rg^9SB;j*wx8V+6@P7FfU`7Z?G1G1vy?Cl3AZnWa&XEw6<8_OFlRFo|U zRS9WIcXMxT&t}@9V$bw2(r-O z6_6RQWk?33^|{N}KpVNf4JE}_N{?N(Sy3Lm#!3$hDO@R$d!6Z`^s;wx;*$GvA-Ka? zLunMtW)eEP7VW}|HtQ5e>1%rvgtH*+=XK5dRcA_$7ru#@qgob~(;{B8Lv??-h5Mns zmw!T0-z(v#X0RcVfa;g|lDM3Xb15{1EBBrS`O-Naw@(e>zgWJFj0nqowyvTpspkYe zijspKuV#bA1_{wO3ni`$uEznWCBmb;#JzaEnRN6OKZ`kKAZ^{KKu#$iFA51D(Y!UsG|<x@PDOazu8`sF!%N!6ET^5zAF`K2vi<09RxK5N9ydh2Q8iV} zGGD$VoHu)a3+5uy`ia?Su9ZPGtA;m>rkw6!a$--miIBt=Cj=FBpcFUz^UhS6$9QP; z;EDLVpnv8)=+|W-zLgz~dKU52Zk6KrGQmi$BO{NwP*+UcHAchAk!j&PdE&o2l5pTrJtHRi)8B*_s=72RqrZO0AE%}?U} zcTM_k^jpq6PDe?_EyQ+yd>(wi&C@di7bAhA_B9Doa+GUrvvW>TpcCsT> z{)}Ewpq~OaTisfe*FTPAm-}j7x^V)BV9P3iYe4AX3w;!$!u|znz++e09@SJuH8wTDl#wcDYSbRnn~y z&*u*W^4x2G?E22uB!I^BnVCHC3mP|&;4s>F!PK>pB)^utA~YCt&@NfX2#7Lkw39VEbOWL!`4nzt6j zNO_mGawBJz-MqZ-V_V_0yI4TbFaMAWYB(t#SnCebxCSlY3-b>;F1Q~%FDWSDFfOS< z(qtu0SgW!`=@ncQIUeND&NH>e0pm(GOa8WuSdAUm3uq>6Xnjv#Patv1-tL#IM6W%3 z-wq$W6`VVAxSq|<=!y3A-!G$42`*{PX%;3tq@=UK=61ZWfnF!l2%qoj0Ib(E(z-e%pR(bZ^i5 zj)1z@8jeH(8giFF$`WY3-Vn#K#rAo#?RxJHHQ*v1`|(Od>2$Hc>#>+6Pcv_%#BqGI z7~%PdxdE|3PV8;8)}!hgZ2-aRdW3Z0nI9 zuD1N<6c@OI%&z%b794opXZva**sw0XEG6sD5dWewt7lL9-j15+9sOo(Ug<(&>T3$; zIO(})d5#r(ua+X6PjN2Pk*2Ojs9lfwb{8u9yib#S`O?L!n>$?K>GFz2C|5dnC@8Q+ z_YC-I@wOOJq`O#QXYP6WporRTnu$+=%RPs=ysyGtBG^^|wFtr?HXGIy%OYG%#ld$A z>DKi%V*!u2PhP*mw;vWo(FZA1lV*b~6csx2O;2Y;+|D);ZjYEwPk1b~K+bCmjdmd_ z6?sB_#^qQdYaNtSO#Ztg&%%7%2vo#x7vZP{e*3w`cend||D(+|WzTkMciryvNtpfV zA)u1Oz=1ao^4m^0>w3wK&lXc4iM(1M1tJd(-y;E-{EuJrabBMN~PzMc^Zp?v7sJ%`mDOpDx?$1m8&7D>R>|fj${X#C+ z^8XCLc?637&Uy;RzWwbdn)D_%N9+Hn?Ye`S`noMh6KOUGMTk-&Dq;vd5l{pXg#dyA z(u?%aLJL(zKxvU8y{I(l2neBr5Ks^hg9?NWp$0+`LQP)q=Qs1dnQz|An>X*y+<#8) zJvV2ceac#U?R&4XEOKBzFfWO#`0V?*;5$u*Z{@DcJ=8n+T*_unb%Segi+t-`Yh$z- z5RZ6Nyda71X?wz9$I!%L(KFz&+83PkbskOK?tZ7zj}$P&%+=vlT2idk>4#?Ay^p!K z^(%jMCt6gY9<5%poUa+hLglM^@^+T9LXv%c&T>S>7F2cRb18>4sXS{^@mv?^SN)N6Ju6is*H~`IlF^B^ znq=i5@#T$~b}}&E?wvKh^_KL?4VyGGSo^+;GWD7LL&WCdw9#CWuT@f{r^0HR3#4M^ zl1qCGz0rqL)|c|*`&z+7m-tEhhVq14183j9iZH;W*2h8iisyG4ed65%PJsj zNetOqS|;$qRIQ6zI>FI}L`I3IM1&>p6K;b0Xm}*P>xZY~ykS|zrgK$+ruTLe z*K~*gIG3Qs>FXOO$7eyP>n~#f(_NLm*mls!Bkbh1x&LL0c9JrDF-xJ)47-z%N#w)d z)M(YdT#tTl4}kG3!Zq^kS|w;GJY!;GIP$==+HCh8Wo zyp56rVtPURoU-~t#Eu!ZN!hjA0uxa2TxWW&)07iia4y=Qpb*VBi) zP7oo+>%uac@)qQ8a@U>VX*dtFK(Y7;sSN;Lx3ZzPdvmO+A@2LOkT3$~P*hUzQRc^DE&`YjeYdW9oM zbWnOQA)M(~sU$q6FG~qlP-4^)d}xgoMtW_`Y6_W;cDuAS@Bfm_8ls!kS4*W# zW20Qsewp4jW(S5GNEMSlbQ3eoJ`-L(9_p%Q4Bl`}iwn`*%v=(~M65Hc=1B8V7H!j3 z`UzDYW9%IBJuTLoRXP#~OGqhKLIGys(fE8<$j;+5sxPG1t?^a=6`ffNH-iZW4tta} zt}~CzHO_clDt??!Sw5F}={|6hujkY-bdlg>0SY@aeuXeH++hgX*f$LEWd5rtW5dA7k!K_8Ta<9$xC!4+UdaMlIx^}!w}Rjk8vm%^tRUAmpA0drkDr=;%^s=&x?}%_m=`Ey zUaYFEW_#)V(UVcFhtm@v6gffjvuy|kKg1{rAP$w;J$v!_{(!-4nudVI6kE(ROQ0_! zws14)ZhO+B_$AJqGKTzdEg|=GdBTUHlaN&6n_)lx$e>+#gqZNg4LvJbWM|_EvhEFS z*L~&06LuHVl_D%r~ z9wC(zF!O+BtT=UoM}tw*Cv09ZUyVw!W3(^?w!6DqV%G3;5o{-x2}jbRB?av%arGUv zN*O=LrtyVMn&KeE9KWd>kF)1@WX7}}Jb;_&`^4XuXFtcK2UBJfbXGA3H0ce`S)Av)zjGGPn3jlz zd|FBh#&NfyR9aE7ZW5Ek_NVjiwF(u@U;$CbnM3udcO5xRkmGRSMzp(d?FaRAi>Xvg z*awsBWP>s*wr{uf6EsbrHGjBYS+F7&!v%e(ekFU zm-2;0{t4#_aKe!|<2_+nhC;khKpu^os(A74-T4Q-45BWr{O?v@-(F2|s?2>@ZXWdO zG0pP3%M%Tzl0v~dQ!vREpOq1&u8*dFhM2MNZA}(h5JR6T>=iZ_Ts+@lDIeiE9xrDU zcLE1!`fxf@S|P`Mz73HiAmW{jodbROoOP4iPH{the5(?_nqL;gF3=hnGd}(c zoXP24V;%Z@J)ZxZFba1~i+>Ldf%+%rXzUk$tans?m0=|^ox8RCYjem3-MbTz+@r3p zj@3p75w2LY5j=Y_2R+wXGH|?^xku)ml?>ip7Y*5~HB{USPFsK3a82c6p?Ib@Au);h zQmK=jI4cc9IOG>qCw~8ao)E(zTJf^Ra6s8dd0*D)jN7$5y9on^L7{ie?^DGuaSoo< zGuKxK4{4pPu-<=v?OL7;Z*JpyYc&(PrK4-e^7r27WTl@PL;9+%L8Y|x8yc%=b0XZ>L)Bz)w?#8RsR(f}9Lj{lpy|Ke3Z8y3nt?IxTU&Vk{X-|Aox0Ue&hd zd7|!Op+wNnL2T`p#YBVnoJ!xx@QNBeQeW0R&71SFI-Eg&zmxyH$j~`#mZaMgKl@8< zMf}~N1TPB7=k7h&OZ#88Z^~YDu9a#!Nz3tKT^8PqKs&xgXPz-v!^p3SY79fg=2?m!a2W5H48*`o4E5n zfmaair-b7|2lJsc^}FtOpI0@lw}#8be0jSo5!bnuqGNsZtkDC*{?9ANkH-yIjg0O) z0x?aMG)Fu0f~?K0Bxe(3lk$Vxk66g;*_b}pR_eu}@j}giP0)!z*D!>y62R4_XEil| z3t|Lbcz-`9w-|X&?EA!XlK%qkDrtU;!l^2xqJ3y);4g7{UFXs=ALumSV>jdIzQgr% zjoW=tq;k8(q3yFdDlMPJ?=`RsXNsw0R8RYdTuo|*=ft47_i)rR?~l??j@@EM)--8*|^pZ ztr}nXrkwU_VhPArtxBZrVPc#b>yO>BT0Q!Cc|9=MCs1nC(B)!!&nvk0FcY$>n1V zOXackJAF3+FzQ!qcK8$kSeJs$vzCUihmwLw0mRX|1n=daADn(%Huql?Yu+CtZTu(? z_8#%3jcY8BG6meA%~X}NpXCrE9knfoh~}+Qw_cT3kM97%SA{*%-Tg=_wEYgDi`*a9 z-}7MOtLQEY92f2dZKoa*L?h4zyWjZ%CfVIN2S1M4NnaX>tvStJ!(i#>i^I7=AW5Q0 z@84AmwleqoNT?>B)o?5?&k?vICeetjd(lIqL4^PW=@I1F5CRZ9uE5Ove)P|G27U5oVm2rv!DO4f>rcs`bbN&;|v@$!~2%=*eVfX z(xj!Wo*ulLUbp_*<#BFR7L@HyUd4fLu$1HoZDtsrZoXC)-QN9aw>)HNRP+kl)W(CI z^RizGk$ijOeU{2;T|Wyp#J9OC?C9|=33yV}e4E9bRfw;2=9d%Q{M+9NQg<5p+QpaC zF6SOr|0Y`t@JzL99^-iH!i>6H$l+CK|8@G~(&_jyz55sBMPAZZTc1YWL&b}kcW_``!T$wjdY*6~3ft>d2Pk&2(#`X6iZRGC#TadK0XPYt~n z0@S;swmmNmfHZDx$}DQWAAuaoo|K2&h4PLG>`l|d*moSELp`wqS76Rh_O@pX$mG)- zH!$nm%=3ax|BzBMaGT>b9F!5KlDP|=3aY~2)Lz284^fDJh^^j`%}Dt^-YRB?%nPoe zj<ymx+P0^6TRrkqLKDg=?Q;08YE+u+MF}(H|Br)H~Np!8q>+^c02TLp=)c zjXPJwHNDSpHfSz88>u~l{;Vv2LN#&pVACl zj6z*<_GAbyV$fWIUKmWcT4vXsj`w+68}F2p*k>db0COGoEd+i60sKwIx_;9*#xTuHCG+s6n&Gr@DtWfzB7Tx-!ZZo|*_ zn}M;8v*XG3 z^O=z=c3e&^e7cxd`BTH!Lh56)Nif0ft-Y^w>4Tdi91L!nY)Y=JW~!-r8bQ6ch$|Da zL8EU$&|_PJ$>_;-Kd_4k9-O^S`mOiSdcf!bdY~nF>?%F~awsE5!jp*(go)PcLUOeh zvs{_49|aMX>$Thi8XTbGS`R7&*}25Dm56C~l`OoE`wqQ>x+8G7`Q8;nMSt;2(!9>kl9j9fp5pg}OCoSS_FVWykfk^v3moZ3(xg zEgn+iCZ}P!`QyOA3@<2$t=p`gu5N22_Z6dBS3wKv-ml>rga8U9&!yN*rR*O}vZ&CV zS5e9S^huSr5sjH373AhJwqt71C(39-VoAjCRpIzse}L+Hv?kB5)J;BRrR=HQu8owb zvR?OHLzm#~mtryZ^2t^kX8~ru6rxB$kQb=k58AG8k%cP5^Y?9ZvKYap=)CQ?kDo3lM8iMDbguX{(-MMrP}?Yz9s_Eg$! z(c^ychClta{FtVnDY0mw?9HtEWsHxHD@gx*$G#skd#tU6b+V7{(Dr6gp> zUh5t0P35l7PKQ}E+|mOGrpPx7_RLsCdKu0ZFSpUKS3jGFMN@!Hhe0<9bo~_(V zLt04wc5D43a7a58k-Hm&47W`h_w}nKdqrDW=C2;I@{TEak*VcBkT-j4z&{Y{OKN2$ zq`SMDXG;+Zg8{tv;ooRN%|FNg3@HB{W6s_<;zSrzA*gI38)0zcZP0~#H~(PHFHgzT ztuxAZK^}D(v1%nzGcMzMvIXr%71j^ZjbK({- zyi}kjboOtnZOzL}>pnG*nxm|E;SrvOz`!+nlNuKfXf)rypd)G+;~}Df7V?Hh8?XCa zhex<;969}xoV2iKKhz%JiBb*H5*Ds(EIHy|tVb8YvqJCigs4Ae`j2q?->q{*q5K8q zcVg?l#q>XRrM2<165_#Acp$CbgOb5r&AB^3iij{-MkM5!#>2-IIFu~}%1TS6s4|rV zTw(iu76Z!UR9}^%wQ4rYFeN!3d~L;x_o(+b^Z!czg*pG_5YynlUi5z`+NWxOzXmY~ U{S>BsDu7N?9j;cUYWeKH07gwlNdN!< literal 0 HcmV?d00001 diff --git a/docs/components.md b/docs/components.md new file mode 100644 index 00000000000..f242eee5938 --- /dev/null +++ b/docs/components.md @@ -0,0 +1,35 @@ + +# Components + +### Core + +Feast Core is the central component that manages Feast and all other components within the system. It allows for the registration and management of entities, features, data stores, and other system resources. Core also manages the execution of feature ingestion jobs from batch and streaming sources, and provides the other Feast components with feature related information. + +### Stores + +Feast maintains data stores for the purposes of model training and serving features to models in production. Features are loaded into these stores by ingestion jobs from both streaming and batch sources. + +Two kinds of data stores are supported: + +Warehouse: The feature warehouse maintains all historical feature data. The warehouse can be queried for batch datasets which are then used for model training. + +Supported warehouse: __BigQuery__ + +Serving: Feast supports multiple serving stores which maintain feature values for access in a production serving environment. + +Supported serving stores: __Redis__, __Bigtable__ + +### Serving + +Feast Serving is an API used for for the retrieval of feature values by models in production. It allows for low latency and high throughput access to feature values from serving stores using Feast client libraries. The API abstracts away data access, allowing users to simultaneously query from multiple stores with a single gRPC or HTTP request. + +### Client Libraries + +Feast provides multiple client libraries for interacting with a Feast deployment. + +| Functionality | CLI | Go | Java | Python (WIP)| +|------------------------------|-----|-----|------|-------------| +| Feature Management | yes | no | no | yes | +| Data Ingestion (Jobs) | yes | no | no | yes | +| Feature Retrieval (Training) | no | no | no | yes | +| Feature Retrieval (Serving) | no | yes | yes | yes | diff --git a/docs/concepts.md b/docs/concepts.md new file mode 100644 index 00000000000..d114d904555 --- /dev/null +++ b/docs/concepts.md @@ -0,0 +1,31 @@ +# Concepts + +### What is Feast? +Feast is a Feature Storage platform for Machine Learning features with the following attributes: + +1. Ingestion and storage of ML features via batch or stream +2. Retrieval of ML features for serving via API, or via Google BigQuery to create training datasets +3. Maintaining of a feature catalog, including additional feature attribute information and discovery via API + +Feast solves a need for standardising how features are stored, served and accessed, and encourages sharing and reuse of created features amongst data science teams. + +Feast does not prescribe how Features should be created. It allows for ingestion via batch or stream in a number of formats, e.g. batch import from CSV, BigQuery tables, streaming via Pub/Sub etc. + + +### What is a Feature? + +A Feature is an individual measurable property or characteristic of an Entity. In the context of Feast a Feature has the following attributes: + +* Entity - It must be associated with a known Entity within Feast +* ValueType - The feature type must be defined, e.g. String, Bytes, Int64, Int32, Float etc. +* Requirements - Properties related to how a feature should be stored for serving and training +* Granularity - Time series features require a defined granularity +* StorageType - For both serving and training a storage type must be defined + +Feast needs to know these attributes in order to be able to ingest, store and serve a feature. A Feature is only a feature when Feast knows about it; This seems contrite, but it introduces a best practice whereby a feature only becomes available for ingestion, serving and training in production when Feast has added the feature to its catalog. + +### What is an Entity? + +An entity is a type with an associated key which generally maps onto a known domain object, e.g. Driver, Customer, Area, Merchant etc. An entity can also be a composite of other entities, with the corresponding composite key, e.g. DriverArea. + +An entity determines how a feature may be retrieved. e.g. for a Driver entity all driver features must be looked up with an associated driver id entity key.