Skip to content

Conversation

@astronautas
Copy link
Contributor

What this PR does / why we need it:

We noticed that for some of our larger get_historical_features jobs, we started getting:

[2025-12-19, 14:16:36 UTC] {pod_manager.py:454} INFO - [base] OperationalError: Error
[2025-12-19, 14:16:36 UTC] {pod_manager.py:454} INFO - [base] HTTPConnectionPool(host='cluster-clickhouse-ml-main.ml.svc.cluster.local',
[2025-12-19, 14:16:36 UTC] {pod_manager.py:454} INFO - [base] port=8123): Read timed out. (read timeout=300) executing HTTP request attempt 1

in essence, we need to be able to control some of the client-side timeouts within the clickhouse client used by the feature store. PR introduces such functionality.

Which issue(s) this PR fixes:

(inline issue within this PR)

@astronautas astronautas requested a review from a team as a code owner December 22, 2025 10:32
@astronautas
Copy link
Contributor Author

astronautas commented Dec 22, 2025

@franciscojavierarceo @ntkathole A small improvement to Clickhouse offline store.

@astronautas astronautas force-pushed the fix/control-clickhouse-offline-store-client-timeouts branch 2 times, most recently from a9c9902 to 508bf0f Compare December 22, 2025 10:44
@astronautas
Copy link
Contributor Author

astronautas commented Dec 22, 2025

I'll additionally test how this more generic additional_args field works with feature_store.yaml config 🕐 . Will let you know in this thread @ntkathole .

@astronautas
Copy link
Contributor Author

astronautas commented Dec 22, 2025

@ntkathole All ready from my end!

@astronautas
Copy link
Contributor Author

astronautas commented Dec 22, 2025

@ntkathole Any ideas?

Run make test-python-integration
python -m pytest --tb=short -v -n 8 --integration --color=yes --durations=10 --timeout=1200 --timeout_method=thread --dist loadgroup \
	-k "(not snowflake or not test_historical_features_main)" \
	-m "not rbac_remote_integration_test" \
	--log-cli-level=INFO -s \
	sdk/python/tests
/opt/hostedtoolcache/Python/3.11.14/x64/bin/python: No module named pytest
make: *** [Makefile:157: test-python-integration] Error 1
Error: Process completed with exit code 2.

can we retry? Unrelated to PR, it seems :/ Caching?

@ntkathole
Copy link
Member

It seems related to aws creds, looking
Error: The security token included in the request is invalid.

Signed-off-by: lukas.valatka <lukas.valatka@cast.ai>
Signed-off-by: lukas.valatka <lukas.valatka@cast.ai>
Signed-off-by: lukas.valatka <lukas.valatka@cast.ai>
Signed-off-by: lukas.valatka <lukas.valatka@cast.ai>
@ntkathole ntkathole force-pushed the fix/control-clickhouse-offline-store-client-timeouts branch from 15c6d7a to a3b9977 Compare December 23, 2025 07:26
Copy link
Member

@ntkathole ntkathole left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good

@ntkathole ntkathole merged commit 59dbb33 into feast-dev:master Dec 23, 2025
17 checks passed
antznette1 pushed a commit to antznette1/feast that referenced this pull request Jan 3, 2026
… offline store (feast-dev#5792)

* add changes + fix linting issues

Signed-off-by: lukas.valatka <lukas.valatka@cast.ai>

* generalize based on pr suggestion

Signed-off-by: lukas.valatka <lukas.valatka@cast.ai>

* add test verifying config parsing

Signed-off-by: lukas.valatka <lukas.valatka@cast.ai>

* rework test into integration, since CH instance is needed

Signed-off-by: lukas.valatka <lukas.valatka@cast.ai>

---------

Signed-off-by: lukas.valatka <lukas.valatka@cast.ai>
Signed-off-by: Anthonette Adanyin <106275232+antznette1@users.noreply.github.com>
franciscojavierarceo pushed a commit that referenced this pull request Jan 16, 2026
# [0.59.0](v0.58.0...v0.59.0) (2026-01-16)

### Bug Fixes

* Add get_table_query_string_with_alias() for PostgreSQL subquery aliasing ([#5811](#5811)) ([11122ce](11122ce))
* Add hybrid online store to ONLINE_STORE_CLASS_FOR_TYPE mapping ([#5810](#5810)) ([678589b](678589b))
* Add possibility to overwrite send_receive_timeout for clickhouse offline store ([#5792](#5792)) ([59dbb33](59dbb33))
* Denial by default to all resources when no permissions set  ([#5663](#5663)) ([1524f1c](1524f1c))
* Make operator include full OIDC secret in repo config ([#5676](#5676)) ([#5809](#5809)) ([a536bc2](a536bc2))
* Populate Postgres `registry.path` during `feast init` ([#5785](#5785)) ([f293ae8](f293ae8))
* **redis:** Preserve millisecond timestamp precision for Redis online store ([#5807](#5807)) ([9e3f213](9e3f213))
* Search API to return all matching tags in matched_tags field ([#5843](#5843)) ([de37f66](de37f66))
* Spark Materialization Engine Cannot Infer Schema ([#5806](#5806)) ([58d0325](58d0325)), closes [#5594](#5594) [#5594](#5594)
* Support arro3 table schema with newer deltalake packages ([#5799](#5799)) ([103c5e9](103c5e9))
* Timestamp formatting and lakehouse-type connector for trino_offline_store. ([#5846](#5846)) ([c2ea7e9](c2ea7e9))
* Update model_validator to use instance method signature (Pydantic v2.12 deprecation) ([#5825](#5825)) ([3c10b6e](3c10b6e))

### Features

* Add dbt integration for importing models as FeatureViews ([#5827](#5827)) ([b997361](b997361)), closes [#3335](#3335) [#3335](#3335) [#3335](#3335)
* Add GCS registry store in Go feature server ([#5818](#5818)) ([1dc2be5](1dc2be5))
* Add progress bar to CLI from feast apply ([#5867](#5867)) ([ab3562b](ab3562b))
* Add RBAC blog post to website ([#5861](#5861)) ([b1844a3](b1844a3))
* Add skip_feature_view_validation parameter to FeatureStore.apply() and plan() ([#5859](#5859)) ([5482a0e](5482a0e))
* Added batching to feature server /push to offline store ([#5683](#5683)) ([#5729](#5729)) ([ce35ce6](ce35ce6))
* Enable static artifacts for feature server that can be used in Feature Transformations ([#5787](#5787)) ([edefc3f](edefc3f))
* Improve lambda materialization engine ([#5829](#5829)) ([f6116f9](f6116f9))
* Offline Store historical features retrieval based on datetime range in Ray ([#5738](#5738)) ([e484c12](e484c12))
* Read, Save docs and chat fixes ([#5865](#5865)) ([2081b55](2081b55))
* Resolve pyarrow >21 installation with ibis-framework ([#5847](#5847)) ([8b9bb50](8b9bb50))
* Support staging for spark materialization ([#5671](#5671)) ([#5797](#5797)) ([5b787af](5b787af))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants