-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Open
Description
Context
PR #5827 added support for FileSource data sources in dbt integration.
Problem
The FileSource implementation generates hardcoded paths that likely don't exist:
# In mapper.py:240-242
return FileSource(
name=f"{model.name}_source",
path=f"/data/{model.name}.parquet", # Hardcoded, likely invalid
# ...
)This path /data/{model.name}.parquet is unlikely to match actual file locations.
Impact
- Generated FileSource objects won't work without manual editing
- Confusing user experience
- No clear documentation on how to fix the paths
Proposed Solutions
Option 1: Make path configurable
feast dbt import -m manifest.json -e driver_id --data-source-type file --file-path-template "/my/data/{model}.parquet"Option 2: Read from dbt meta
Support custom path in dbt model meta:
# dbt model
models:
- name: driver_stats
meta:
feast:
file_path: "/data/warehouse/driver_stats.parquet"Option 3: Require explicit path mapping file
feast dbt import -m manifest.json -e driver_id --file-path-config paths.yamlRecommendation
Implement Option 1 (path template) as it's simplest and covers most use cases. Consider Option 2 for advanced scenarios.
Related
Metadata
Metadata
Assignees
Labels
No labels