feat(v2): add from and to pandas df for documentarray #1161
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai>
Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai>
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
added 3 commits
February 22, 2023 11:50
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
…to-pandas Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
47 tasks
samsja
reviewed
Feb 23, 2023
samsja
reviewed
Feb 23, 2023
pyproject.toml
Outdated
| rich = ">=13.1.0" | ||
| lz4 = {version= ">=1.0.0", optional = true} | ||
| pydub = {version = "^0.25.1", optional = true } | ||
| pandas = ">=1.1.0" |
Member
There was a problem hiding this comment.
it should be an optional dependecy
Contributor
Author
There was a problem hiding this comment.
add it to common or a as a separate extra pandas ?
Member
There was a problem hiding this comment.
separate. we will reorginaze at some point anyway
samsja
reviewed
Feb 23, 2023
Member
samsja
left a comment
There was a problem hiding this comment.
look goods, I added some comments
JohannesMessner
requested changes
Feb 23, 2023
docarray/array/array/io.py
Outdated
| ) | ||
|
|
||
| @staticmethod | ||
| def access_path_dict_to_nested_dict( |
Member
There was a problem hiding this comment.
didn't you already implement something like this in your previous PR? or am i mixing things up?
Contributor
Author
There was a problem hiding this comment.
yes, the function _access_path_to_dict that is used within _access_path_dict_to_nested_dict that I implemented in the last PR.
_access_path_to_dict: transforms one access path to nested dict_ access_path_dict_to_nested_dict: transforms dict with (multiple) access path keys to a joint nested dict by calling the former func
I will move the latter to that other helper file, to keep those functions together yes?
Member
There was a problem hiding this comment.
ok sounds good, yes keeping them in the same helper file would be ideal
docarray/array/array/io.py
Outdated
| return json.dumps([doc.json() for doc in self]) | ||
|
|
||
| @classmethod | ||
| def _check_for_valid_document_type(cls) -> None: |
Member
There was a problem hiding this comment.
I would prefer returning a bool and rasing at the call site. That way it is clearer where the error actually occurs, and this method can be re-used in contexts where only the information is needed, without wanting to raise an exception
docarray/array/array/io.py
Outdated
| ) | ||
|
|
||
| @classmethod | ||
| def _check_for_valid_access_paths(cls, field_names: Optional[List[str]]) -> None: |
docarray/array/array/io.py
Outdated
|
|
||
| @classmethod | ||
| def _check_for_valid_document_type(cls) -> None: | ||
| if cls.document_type == AnyDocument: |
Member
There was a problem hiding this comment.
is there are plan to make this check more sophisticated as part of this PR? otherwise this doesn't need to be a method imo
Contributor
Author
There was a problem hiding this comment.
No, just tried to extract all the duplicate code but maybe I went a bit overboard hehe, moved it back to from_csv and from_pandas again
This was
linked to
issues
Feb 23, 2023
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai>
Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai>
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
samsja
reviewed
Feb 23, 2023
samsja
reviewed
Feb 23, 2023
samsja
requested changes
Feb 23, 2023
Member
samsja
left a comment
There was a problem hiding this comment.
we should make pandas install optional. Otherwise looks good
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
samsja
approved these changes
Feb 23, 2023
|
📝 Docs are deployed on https://ft-feat-from-to-pandas--jina-docs.netlify.app 🎉 |
JohannesMessner
approved these changes
Feb 23, 2023
AsRaNi1
pushed a commit
to AsRaNi1/docarray
that referenced
this pull request
Feb 28, 2023
* feat: load from and to csv Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: from to csv Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * feat: add access path to dict Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: from to csv Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: clean up Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * docs: add docstring and update tmpdir in test Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: merge nested dicts Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: clean up Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: clean up Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * test: update test Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: apply samis suggestion from code review Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: apply suggestions from code review wrt access paths Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: apply johannes suggestion Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai> * fix: apply johannes suggestion Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai> * fix: apply suggestions from code review Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: apply suggestions from code review Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: typos Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * refactor: move helper functions to helper file Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * test: fix fixture Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * feat: add to and from pandas df for documentarray Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * chore: add pandas to pyproject.toml Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * docs: update docstring Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: mypy Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: clean up Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: apply suggestions from code review Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: apply suggestion from johannes Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai> * fix: apply suggestion from johannes Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai> * fix: apply suggestions from code review Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: apply suggestion Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: apply suggestions Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> --------- Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> Signed-off-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai> Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: Arnav Zutshi <arnzut1324@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add from and to pandas to DocumentArray IOMixin
DocumentArray[Doc].from_pandas(df)da.to_pandas()