From d1c8ed5269c2ca55eb4633aa4ec391ecbdb4d502 Mon Sep 17 00:00:00 2001 From: Pushkar Gupta Date: Wed, 24 Apr 2024 17:14:04 -0700 Subject: [PATCH 1/4] feat: Feast/IKV online store documentation Signed-off-by: Pushkar Gupta --- README.md | 1 + docs/SUMMARY.md | 1 + docs/reference/online-stores/README.md | 4 ++ docs/reference/online-stores/ikv.md | 95 ++++++++++++++++++++++++++ docs/roadmap.md | 1 + 5 files changed, 102 insertions(+) create mode 100644 docs/reference/online-stores/ikv.md diff --git a/README.md b/README.md index 6a851d0d417..aab9c332435 100644 --- a/README.md +++ b/README.md @@ -175,6 +175,7 @@ The list below contains the functionality that contributors are planning to deve * [x] [Bigtable](https://docs.feast.dev/reference/online-stores/bigtable) * [x] [SQLite](https://docs.feast.dev/reference/online-stores/sqlite) * [x] [Dragonfly](https://docs.feast.dev/reference/online-stores/dragonfly) + * [x] [IKV - Inlined Key Value Store](https://docs.feast.dev/reference/online-stores/ikv) * [x] [Azure Cache for Redis (community plugin)](https://github.com/Azure/feast-azure) * [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/online-stores/postgres) * [x] [Cassandra / AstraDB (contrib plugin)](https://docs.feast.dev/reference/online-stores/cassandra) diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 8affea898ef..b211730d0ef 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -91,6 +91,7 @@ * [Snowflake](reference/online-stores/snowflake.md) * [Redis](reference/online-stores/redis.md) * [Dragonfly](reference/online-stores/dragonfly.md) + * [IKV](reference/online-stores/ikv.md) * [Datastore](reference/online-stores/datastore.md) * [DynamoDB](reference/online-stores/dynamodb.md) * [Bigtable](reference/online-stores/bigtable.md) diff --git a/docs/reference/online-stores/README.md b/docs/reference/online-stores/README.md index d90bfcf1632..686e820f4e7 100644 --- a/docs/reference/online-stores/README.md +++ b/docs/reference/online-stores/README.md @@ -22,6 +22,10 @@ Please see [Online Store](../../getting-started/architecture-and-components/onli [dragonfly.md](dragonfly.md) {% endcontent-ref %} +{% content-ref url="ikv.md" %} +[ikv.md](ikv.md) +{% endcontent-ref %} + {% content-ref url="datastore.md" %} [datastore.md](datastore.md) {% endcontent-ref %} diff --git a/docs/reference/online-stores/ikv.md b/docs/reference/online-stores/ikv.md new file mode 100644 index 00000000000..911ce97798d --- /dev/null +++ b/docs/reference/online-stores/ikv.md @@ -0,0 +1,95 @@ +# IKV (Inlined Key-Value Store) online store + +## Description + +[IKV](https://github.com/inlinedio/ikv-store) is a fully-managed embedded key-value store, primarily designed for storing ML features. Most key-value stores (think Redis or Cassandra) need a remote database cluster, whereas IKV allows you to utilize your existing application infrastructure to store data (cost efficient) and access it without any network calls (better performance). + +For provisioning API keys for using it as an online-store in Feast, go to [https://inlined.io](https://inlined.io) + +## Getting started + +Make sure you have Python and `pip` installed. + +Install the Feast SDK and CLI + +`pip install feast` + +In order to use IKV as the online store, you'll need to install the extra: + +`pip install 'feast[ikv]'` + +### 1. Provision an IKV store +Go to [https://inlined.io](https://inlined.io) or email onboarding[at]inlined.io +IKV does not support docker deployment for local testing at the moment. + +### 2. Create a feature repository + +Bootstrap a new feature repository: + +``` +feast init my_feature_repo +cd my_feature_repo/feature_repo +``` + +Update `my_feature_repo/feature_store.yaml` with the below contents: + +{% code title="feature_store.yaml" %} +```yaml +project: my_feature_repo +registry: data/registry.db +provider: local +online_store: + type: ikv + account_id: secret + account_passkey: secret + store_name: your-store-name + mount_directory: /absolute/path/on/disk/for/ikv/embedded/index +``` +{% endcode %} + +The full set of configuration options is available in IKVOnlineStoreConfig at `sdk/python/feast/infra/online_stores/contrib/ikv_online_store/ikv.py` + +### 3. Register feature definitions and deploy your feature store + +`feast apply` + +The `apply` command scans python files in the current directory (`example_repo.py` in this case) for feature view/entity definitions, registers the objects, and deploys infrastructure. +You should see the following output: + +``` +.... +Created entity driver +Created feature view driver_hourly_stats_fresh +Created feature view driver_hourly_stats +Created on demand feature view transformed_conv_rate +Created on demand feature view transformed_conv_rate_fresh +Created feature service driver_activity_v1 +Created feature service driver_activity_v3 +Created feature service driver_activity_v2 +``` + +## Functionality Matrix + +The set of functionality supported by online stores is described in detail [here](overview.md#functionality). +Below is a matrix indicating which functionality is supported by the Redis online store. + +| | Redis | +| :-------------------------------------------------------- | :---- | +| write feature values to the online store | yes | +| read feature values from the online store | yes | +| update infrastructure (e.g. tables) in the online store | yes | +| teardown infrastructure (e.g. tables) in the online store | yes | +| generate a plan of infrastructure changes | no | +| support for on-demand transforms | yes | +| readable by Python SDK | yes | +| readable by Java | yes | +| readable by Go | yes | +| support for entityless feature views | yes | +| support for concurrent writing to the same key | yes | +| support for ttl (time to live) at retrieval | yes | +| support for deleting expired data | yes | +| collocated by feature view | no | +| collocated by feature service | no | +| collocated by entity key | yes | + +To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix). diff --git a/docs/roadmap.md b/docs/roadmap.md index a04ede7c993..5ff262e3432 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -33,6 +33,7 @@ The list below contains the functionality that contributors are planning to deve * [x] [Bigtable](https://docs.feast.dev/reference/online-stores/bigtable) * [x] [SQLite](https://docs.feast.dev/reference/online-stores/sqlite) * [x] [Dragonfly](https://docs.feast.dev/reference/online-stores/dragonfly) + * [x] [IKV - Inlined Key Value Store](https://docs.feast.dev/reference/online-stores/ikv) * [x] [Azure Cache for Redis (community plugin)](https://github.com/Azure/feast-azure) * [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/online-stores/postgres) * [x] [Cassandra / AstraDB (contrib plugin)](https://docs.feast.dev/reference/online-stores/cassandra) From 4816f4885c14b607f8a4189f1010636d4f94223b Mon Sep 17 00:00:00 2001 From: Pushkar Gupta Date: Wed, 24 Apr 2024 17:38:41 -0700 Subject: [PATCH 2/4] functionality matric Signed-off-by: Pushkar Gupta --- docs/reference/online-stores/ikv.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/reference/online-stores/ikv.md b/docs/reference/online-stores/ikv.md index 911ce97798d..23db7933ba8 100644 --- a/docs/reference/online-stores/ikv.md +++ b/docs/reference/online-stores/ikv.md @@ -71,9 +71,9 @@ Created feature service driver_activity_v2 ## Functionality Matrix The set of functionality supported by online stores is described in detail [here](overview.md#functionality). -Below is a matrix indicating which functionality is supported by the Redis online store. +Below is a matrix indicating which functionality is supported by the IKV online store. -| | Redis | +| | IKV | | :-------------------------------------------------------- | :---- | | write feature values to the online store | yes | | read feature values from the online store | yes | @@ -82,14 +82,14 @@ Below is a matrix indicating which functionality is supported by the Redis onlin | generate a plan of infrastructure changes | no | | support for on-demand transforms | yes | | readable by Python SDK | yes | -| readable by Java | yes | -| readable by Go | yes | +| readable by Java | no | +| readable by Go | no | | support for entityless feature views | yes | | support for concurrent writing to the same key | yes | -| support for ttl (time to live) at retrieval | yes | -| support for deleting expired data | yes | -| collocated by feature view | no | -| collocated by feature service | no | +| support for ttl (time to live) at retrieval | no | +| support for deleting expired data | no | +| collocated by feature view | yes | +| collocated by feature service | ?? | | collocated by entity key | yes | To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix). From 40cd2dcf2412f9a54389bf21d5013f94b6afdf8f Mon Sep 17 00:00:00 2001 From: Pushkar Gupta Date: Wed, 24 Apr 2024 21:57:42 -0700 Subject: [PATCH 3/4] more changes Signed-off-by: Pushkar Gupta --- docs/reference/online-stores/ikv.md | 20 ++++++------- docs/reference/online-stores/overview.md | 38 ++++++++++++------------ 2 files changed, 29 insertions(+), 29 deletions(-) diff --git a/docs/reference/online-stores/ikv.md b/docs/reference/online-stores/ikv.md index 23db7933ba8..3c6c798c98f 100644 --- a/docs/reference/online-stores/ikv.md +++ b/docs/reference/online-stores/ikv.md @@ -4,23 +4,23 @@ [IKV](https://github.com/inlinedio/ikv-store) is a fully-managed embedded key-value store, primarily designed for storing ML features. Most key-value stores (think Redis or Cassandra) need a remote database cluster, whereas IKV allows you to utilize your existing application infrastructure to store data (cost efficient) and access it without any network calls (better performance). -For provisioning API keys for using it as an online-store in Feast, go to [https://inlined.io](https://inlined.io) +For provisioning API keys for using it as an online-store in Feast, go to [https://inlined.io](https://inlined.io) or email onboarding[at]inlined.io ## Getting started - Make sure you have Python and `pip` installed. -Install the Feast SDK and CLI - -`pip install feast` +Install the Feast SDK and CLI: `pip install feast` -In order to use IKV as the online store, you'll need to install the extra: +In order to use this online store, you'll need to install the IKV extra (along with the dependency needed for the offline store of choice). E.g. +- `pip install 'feast[gcp, ikv]'` +- `pip install 'feast[snowflake, ikv]'` +- `pip install 'feast[aws, ikv]'` +- `pip install 'feast[azure, ikv]'` -`pip install 'feast[ikv]'` +You can get started by using any of the other templates (e.g. `feast init -t gcp` or `feast init -t snowflake` or `feast init -t aws`), and then swapping in IKV as the online store as seen below in the examples. ### 1. Provision an IKV store Go to [https://inlined.io](https://inlined.io) or email onboarding[at]inlined.io -IKV does not support docker deployment for local testing at the moment. ### 2. Create a feature repository @@ -88,8 +88,8 @@ Below is a matrix indicating which functionality is supported by the IKV online | support for concurrent writing to the same key | yes | | support for ttl (time to live) at retrieval | no | | support for deleting expired data | no | -| collocated by feature view | yes | -| collocated by feature service | ?? | +| collocated by feature view | no | +| collocated by feature service | no | | collocated by entity key | yes | To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix). diff --git a/docs/reference/online-stores/overview.md b/docs/reference/online-stores/overview.md index 7a51a9a4687..04d24447058 100644 --- a/docs/reference/online-stores/overview.md +++ b/docs/reference/online-stores/overview.md @@ -29,26 +29,26 @@ See this [issue](https://github.com/feast-dev/feast/issues/2254) for a discussio ## Functionality Matrix There are currently five core online store implementations: `SqliteOnlineStore`, `RedisOnlineStore`, `DynamoDBOnlineStore`, `SnowflakeOnlineStore`, and `DatastoreOnlineStore`. -There are several additional implementations contributed by the Feast community (`PostgreSQLOnlineStore`, `HbaseOnlineStore`, and `CassandraOnlineStore`), which are not guaranteed to be stable or to match the functionality of the core implementations. +There are several additional implementations contributed by the Feast community (`PostgreSQLOnlineStore`, `HbaseOnlineStore`, `CassandraOnlineStore` and `IKVOnlineStore`), which are not guaranteed to be stable or to match the functionality of the core implementations. Details for each specific online store, such as how to configure it in a `feature_store.yaml`, can be found [here](README.md). Below is a matrix indicating which online stores support what functionality. -| | Sqlite | Redis | DynamoDB | Snowflake | Datastore | Postgres | Hbase | [[Cassandra](https://cassandra.apache.org/_/index.html) / [Astra DB](https://www.datastax.com/products/datastax-astra?utm_source=feast)] | -| :-------------------------------------------------------- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | -| write feature values to the online store | yes | yes | yes | yes | yes | yes | yes | yes | -| read feature values from the online store | yes | yes | yes | yes | yes | yes | yes | yes | -| update infrastructure (e.g. tables) in the online store | yes | yes | yes | yes | yes | yes | yes | yes | -| teardown infrastructure (e.g. tables) in the online store | yes | yes | yes | yes | yes | yes | yes | yes | -| generate a plan of infrastructure changes | yes | no | no | no | no | no | no | yes | -| support for on-demand transforms | yes | yes | yes | yes | yes | yes | yes | yes | -| readable by Python SDK | yes | yes | yes | yes | yes | yes | yes | yes | -| readable by Java | no | yes | no | no | no | no | no | no | -| readable by Go | yes | yes | no | no | no | no | no | no | -| support for entityless feature views | yes | yes | yes | yes | yes | yes | yes | yes | -| support for concurrent writing to the same key | no | yes | no | no | no | no | no | no | -| support for ttl (time to live) at retrieval | no | yes | no | no | no | no | no | no | -| support for deleting expired data | no | yes | no | no | no | no | no | no | -| collocated by feature view | yes | no | yes | yes | yes | yes | yes | yes | -| collocated by feature service | no | no | no | no | no | no | no | no | -| collocated by entity key | no | yes | no | no | no | no | no | no | +| | Sqlite | Redis | DynamoDB | Snowflake | Datastore | Postgres | Hbase | [[Cassandra](https://cassandra.apache.org/_/index.html) / [Astra DB](https://www.datastax.com/products/datastax-astra?utm_source=feast)] | [IKV](https://inlined.io) | +| :-------------------------------------------------------- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- | +| write feature values to the online store | yes | yes | yes | yes | yes | yes | yes | yes | yes | +| read feature values from the online store | yes | yes | yes | yes | yes | yes | yes | yes | yes | +| update infrastructure (e.g. tables) in the online store | yes | yes | yes | yes | yes | yes | yes | yes | yes | +| teardown infrastructure (e.g. tables) in the online store | yes | yes | yes | yes | yes | yes | yes | yes | yes | +| generate a plan of infrastructure changes | yes | no | no | no | no | no | no | yes | no | +| support for on-demand transforms | yes | yes | yes | yes | yes | yes | yes | yes | yes | +| readable by Python SDK | yes | yes | yes | yes | yes | yes | yes | yes | yes | +| readable by Java | no | yes | no | no | no | no | no | no | no | +| readable by Go | yes | yes | no | no | no | no | no | no | no | +| support for entityless feature views | yes | yes | yes | yes | yes | yes | yes | yes | yes | +| support for concurrent writing to the same key | no | yes | no | no | no | no | no | no | yes | +| support for ttl (time to live) at retrieval | no | yes | no | no | no | no | no | no | no | +| support for deleting expired data | no | yes | no | no | no | no | no | no | no | +| collocated by feature view | yes | no | yes | yes | yes | yes | yes | yes | no | +| collocated by feature service | no | no | no | no | no | no | no | no | no | +| collocated by entity key | no | yes | no | no | no | no | no | no | yes | From 20ae9b1879503bc61e8d7d6e9630c1b63ca235c0 Mon Sep 17 00:00:00 2001 From: Pushkar Gupta Date: Wed, 24 Apr 2024 22:05:35 -0700 Subject: [PATCH 4/4] mount dir Signed-off-by: Pushkar Gupta --- docs/reference/online-stores/ikv.md | 30 ++++------------------------- 1 file changed, 4 insertions(+), 26 deletions(-) diff --git a/docs/reference/online-stores/ikv.md b/docs/reference/online-stores/ikv.md index 3c6c798c98f..ff690c1a622 100644 --- a/docs/reference/online-stores/ikv.md +++ b/docs/reference/online-stores/ikv.md @@ -22,14 +22,7 @@ You can get started by using any of the other templates (e.g. `feast init -t gcp ### 1. Provision an IKV store Go to [https://inlined.io](https://inlined.io) or email onboarding[at]inlined.io -### 2. Create a feature repository - -Bootstrap a new feature repository: - -``` -feast init my_feature_repo -cd my_feature_repo/feature_repo -``` +### 2. Configure Update `my_feature_repo/feature_store.yaml` with the below contents: @@ -47,26 +40,11 @@ online_store: ``` {% endcode %} -The full set of configuration options is available in IKVOnlineStoreConfig at `sdk/python/feast/infra/online_stores/contrib/ikv_online_store/ikv.py` - -### 3. Register feature definitions and deploy your feature store - -`feast apply` +After provision an IKV account/store, you should the required id, passkey and store-name. -The `apply` command scans python files in the current directory (`example_repo.py` in this case) for feature view/entity definitions, registers the objects, and deploys infrastructure. -You should see the following output: +Additionally you must specify a mount-directory - where IKV will pull/update (maintain) a copy of the index for online reads (IKV is an embedded database). It can be skipped only if you don't plan to read any data from this container. The mount directory path usually points to a location on local/remote disk. -``` -.... -Created entity driver -Created feature view driver_hourly_stats_fresh -Created feature view driver_hourly_stats -Created on demand feature view transformed_conv_rate -Created on demand feature view transformed_conv_rate_fresh -Created feature service driver_activity_v1 -Created feature service driver_activity_v3 -Created feature service driver_activity_v2 -``` +The full set of configuration options is available in IKVOnlineStoreConfig at `sdk/python/feast/infra/online_stores/contrib/ikv_online_store/ikv.py` ## Functionality Matrix