Implemented get_historical_features() with job.persist() using a Spark offline store. However, job.persist() failed with a ValueError stating the method does not support SparkSource. Found that SavedDatasetStorage.from_data_source() only maps to FileSource. When I attempted to use a FileSource, it failed with an assertion error because SparkRetrievalJob.persist specifically expects SavedDatasetSparkStorage.
Additionally, We are using the Spark offline store with a path based SparkSource configuration (S3 with parquet format). When I went through the code found that the method only supports table based SparkSource, can you please consider adding support for path based spark source configuration as well?