Skip to content

feat: tensor type for protobuf deserialization #1645

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
JoanFM merged 9 commits into from
Jun 14, 2023
Merged

feat: tensor type for protobuf deserialization #1645

JoanFM merged 9 commits into from
Jun 14, 2023

Conversation

@JohannesMessner
Copy link
Member

@JohannesMessner JohannesMessner commented Jun 13, 2023 •
edited
Loading

This allows DocVoc to be deserilzed to a specific tensor_type, i.e. torch, tf, or numpy:

# look at the unit test for more comprehensive example
class MyDoc(BaseDoc):
    tensor: TensorFlowTensor

da = DocVec[MyDoc](...)  # doesn't matter what tensor_type is here

proto = da.to_protobuf()
da_after = DocVec[MyDoc].from_protobuf(proto, tensor_type=TensorFlowTensor)

assert isinstance(da_after.tensor, TensorFlowTensor)

Note that the tensor_type passed to from_protobuf() does not need to match the tensor type from before serialization.
Since all tensor are represented the same way in the proto, any proto can be deserialized to any tensor type.

TODO:

  • docs

@JohannesMessner
feat: tensor type for protobuf
e70f033 
Signed-off-by: Johannes Messner <messnerjo@gmail.com>
@JohannesMessner JohannesMessner mentioned this pull request Jun 13, 2023
Merged
9 tasks
@JohannesMessner
docs: explain tensor type from protobuf
ddd7b19 
Signed-off-by: Johannes Messner <messnerjo@gmail.com>
@JohannesMessner
test: fix tf tensor import
eaf2a0f 
Signed-off-by: Johannes Messner <messnerjo@gmail.com>
@JohannesMessner JohannesMessner requested a review from samsja June 13, 2023 14:05
samsja
samsja reviewed Jun 13, 2023
View reviewed changes
Copy link
Member

@samsja samsja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so what happened if I do to_profobuf with a TorchTensor in my doc_type ? It will be loaded as numpy tensor right ?

@JohannesMessner
test: try to fix tf test
8a32574 
Signed-off-by: Johannes Messner <messnerjo@gmail.com>
@JohannesMessner
Copy link
Member Author

JohannesMessner commented Jun 13, 2023

so what happened if I do to_profobuf with a TorchTensor in my doc_type ? It will be loaded as numpy tensor right ?

yes, if not tensor_type is specified in from_protobuf it will attempt to load it as numpy. This mirrors the behaviour of DocVec.__init__().

@JohannesMessner
test: another go at the tf test
ae65251 
Signed-off-by: Johannes Messner <messnerjo@gmail.com>
@JohannesMessner
test: another go at the tf test
76d4fc8 
Signed-off-by: Johannes Messner <messnerjo@gmail.com>
@JohannesMessner
test: another go at the tf test
026a8ef 
Signed-off-by: Johannes Messner <messnerjo@gmail.com>
@JohannesMessner
test: fing tf pls work now
c2812cf 
Signed-off-by: Johannes Messner <messnerjo@gmail.com>
@JohannesMessner JohannesMessner marked this pull request as ready for review June 13, 2023 15:28
@JohannesMessner
test: remove proto mark
f583129 
Signed-off-by: Johannes Messner <messnerjo@gmail.com>
@github-actions
Copy link

github-actions bot commented Jun 13, 2023

📝 Docs are deployed on https://ft-feat-protobuf-tensor-type--jina-docs.netlify.app 🎉

@JohannesMessner JohannesMessner requested a review from samsja June 14, 2023 07:19
@JoanFM JoanFM merged commit eb3f857 into main Jun 14, 2023
@JoanFM JoanFM deleted the feat-protobuf-tensor-type branch June 14, 2023 09:23
@samsja samsja mentioned this pull request Jun 19, 2023
Closed
@scott-martens scott-martens mentioned this pull request Jun 20, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@samsja samsja samsja approved these changes

Assignees

No one assigned

Labels

area/core area/docs area/testing component/array size/m

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@JohannesMessner @samsja @JoanFM