Skip to content

fix: Fix type mapping spark #4071

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
HaoXuAI merged 8 commits into from
Apr 15, 2024
Merged

fix: Fix type mapping spark #4071

Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
2f37e07
feat: Enable Arrow-based columnar data transfers
ElliotNguyen68 Mar 19, 2024
8eb0c59
Merge branch 'feast-dev:master' into master
ElliotNguyen68 Mar 21, 2024
c1c9990
fix: Add __eq__, __hash__ to SparkSource for comparision
ElliotNguyen68 Mar 21, 2024
add7f3a
chore: simplify the logic
ElliotNguyen68 Mar 21, 2024
6cba191
Merge branch 'feast-dev:master' into master
ElliotNguyen68 Mar 23, 2024
21e5434
Merge branch 'feast-dev:master' into master
ElliotNguyen68 Apr 3, 2024
5bc1ec6
feat: Add decimal datatype to spark mapping to feast data type
ElliotNguyen68 Apr 8, 2024
8dc98b3
feat: add array decimal
ElliotNguyen68 Apr 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion sdk/python/feast/type_map.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -752,7 +752,7 @@ def _non_empty_value(value: Any) -> bool:

def spark_to_feast_value_type(spark_type_as_str: str) -> ValueType:
# TODO not all spark types are convertible
# Current non-convertible types: interval, map, struct, structfield, decimal, binary
# Current non-convertible types: interval, map, struct, structfield, binary
type_map: Dict[str, ValueType] = {
"null": ValueType.UNKNOWN,
"byte": ValueType.BYTES,
Expand All @@ -762,6 +762,7 @@ def spark_to_feast_value_type(spark_type_as_str: str) -> ValueType:
"bigint": ValueType.INT64,
"long": ValueType.INT64,
"double": ValueType.DOUBLE,
"decimal": ValueType.DOUBLE,
"float": ValueType.FLOAT,
"boolean": ValueType.BOOL,
"timestamp": ValueType.UNIX_TIMESTAMP,
Expand All @@ -770,10 +771,15 @@ def spark_to_feast_value_type(spark_type_as_str: str) -> ValueType:
"array<int>": ValueType.INT32_LIST,
"array<bigint>": ValueType.INT64_LIST,
"array<double>": ValueType.DOUBLE_LIST,
"array<decimal>": ValueType.DOUBLE_LIST,
"array<float>": ValueType.FLOAT_LIST,
"array<boolean>": ValueType.BOOL_LIST,
"array<timestamp>": ValueType.UNIX_TIMESTAMP_LIST,
}
if spark_type_as_str.startswith("decimal"):
spark_type_as_str = "decimal"
if spark_type_as_str.startswith("array<decimal"):
spark_type_as_str = "array<decimal>"
# TODO: Find better way of doing this.
if not isinstance(spark_type_as_str, str) or spark_type_as_str not in type_map:
return ValueType.NULL
Expand Down