Skip to content

dbt integration: Add support for data source connection configuration #5875

@YassinNouh21

Description

@YassinNouh21

Context

PR #5827 added dbt integration with basic data source support.

Problem

The generated data sources use default/inferred configurations but don't allow specifying:

  • BigQuery: GCP project, dataset overrides
  • Snowflake: account, warehouse, role, connection parameters
  • File: base path, file format options

This means generated sources may not connect to the actual data.

Current Limitation

# BigQuery example - no way to override project
return BigQuerySource(
    name=f"{model.name}_source",
    table=model.full_table_name,  # Uses dbt's database.schema.table
    # Missing: project override, dataset override
)

Proposed Enhancement

Add CLI options for data source configuration:

# BigQuery
feast dbt import -m manifest.json -e driver_id \
  --data-source-type bigquery \
  --bigquery-project my-gcp-project

# Snowflake  
feast dbt import -m manifest.json -e driver_id \
  --data-source-type snowflake \
  --snowflake-account my-account \
  --snowflake-warehouse my-warehouse \
  --snowflake-role my-role

# File
feast dbt import -m manifest.json -e driver_id \
  --data-source-type file \
  --file-format parquet \
  --file-base-path /data/warehouse

Or support reading from feature_store.yaml:

offline_store:
  type: bigquery
  project: my-gcp-project
  
dbt:
  inherit_offline_store_config: true

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions