feat: nested attribute access in `find()` #1176

anna-charlotte · 2023-02-24T11:15:38Z

Goals:
allow nested attribute access in find(), which currently isn't supported yet. Nested access will be possible by '__'-separated access paths.

Example:

class InnerDoc(BaseDocument):
    title: str
    embedding: NdArray

class MyDoc(BaseDocument):
    inner: InnerDoc

query = MyDoc(inner=InnerDoc(title='query', embedding=np.random.randn((10))))
index = DocumentArray[MyDoc](
    [
        MyDoc(inner=InnerDoc(title=f'doc {i}', embedding=np.random.randn((10))))
        for i in range(10)
    ]
)

top_k, scores = find(
    index,
    query,
    embedding_field='inner__embedding',  # <----------- access path for nested access
    limit=7,
)

check and update documentation, if required. See guide

docarray/utils/find.py

docarray/helper.py

samsja · 2023-02-27T09:04:25Z

docarray/utils/find.py

+        emb = embedding_type._docarray_stack(emb_list)
    elif isinstance(data, DocumentArrayStacked):
-        emb = getattr(data, embedding_field)
+        emb = next(AnyDocumentArray._traverse(data, embedding_field))


not sure to understand why it is next here

For DocumentArray stacked, the _traverse will yield one stacked tensor, by calling next() on it, I want to get that one tensor. I think this would be the same as calling list(AnyDocumentArray._traverse(data, embedding_field))[0].

JohannesMessner · 2023-02-27T13:30:00Z

docarray/helper.py

+
+def _get_field_type_by_access_path(
+    doc_type: Type['BaseDocument'], access_path: str
+) -> Any:


-> Optional[Type]?

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

github-actions · 2023-02-27T13:50:46Z

📝 Docs are deployed on https://ft-feat-find-nested-access--jina-docs.netlify.app 🎉

* test: add test for find on nested attribute Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * test: add test for nested attr in stacked da Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: nested access Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: mypy Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: field type by access path for da Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: clean up Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: move get field type by access path Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: apply suggestion and fix import Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: apply suggestion from code review Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: apply suggestion from code review Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: apply suggestions Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> --------- Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> Signed-off-by: Arnav Zutshi <arnzut1324@gmail.com>

* test: add test for find on nested attribute Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * test: add test for nested attr in stacked da Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: nested access Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: mypy Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: field type by access path for da Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: clean up Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: move get field type by access path Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: apply suggestion and fix import Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: apply suggestion from code review Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: apply suggestion from code review Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: apply suggestions Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> --------- Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

github-actions bot added size/s area/testing labels Feb 24, 2023

anna-charlotte mentioned this pull request Feb 24, 2023

Meta: DocArray v2 Roadmap #780

Closed

47 tasks

github-actions bot added area/core size/m and removed size/s labels Feb 24, 2023

anna-charlotte marked this pull request as ready for review February 27, 2023 08:40

JohannesMessner reviewed Feb 27, 2023

View reviewed changes

docarray/utils/find.py Outdated Show resolved Hide resolved

samsja reviewed Feb 27, 2023

View reviewed changes

JohannesMessner requested changes Feb 27, 2023

View reviewed changes

anna-charlotte added 12 commits February 27, 2023 14:45

test: add test for find on nested attribute

d1f651b

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

test: add test for nested attr in stacked da

e7c4178

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

fix: nested access

5ba1857

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

fix: mypy

7c9a617

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

fix: field type by access path for da

5c7e862

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

fix: clean up

e8a0003

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

fix: move get field type by access path

8ac4827

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

fix: imports

2cc5a21

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

fix: apply suggestion and fix import

234cb36

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

fix: apply suggestion from code review

95a0bd0

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

fix: apply suggestion from code review

c520630

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

fix: apply suggestions

8400ed4

Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>

anna-charlotte force-pushed the feat-find-nested-access branch from 68af091 to 8400ed4 Compare February 27, 2023 13:46

samsja approved these changes Feb 27, 2023

View reviewed changes

JohannesMessner approved these changes Feb 27, 2023

View reviewed changes

anna-charlotte merged commit 28b96fe into feat-rewrite-v2 Feb 27, 2023

anna-charlotte deleted the feat-find-nested-access branch February 27, 2023 16:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: nested attribute access in `find()` #1176

feat: nested attribute access in `find()` #1176

Uh oh!

anna-charlotte commented Feb 24, 2023

Uh oh!

Uh oh!

Uh oh!

samsja Feb 27, 2023

Uh oh!

anna-charlotte Feb 27, 2023

Uh oh!

JohannesMessner Feb 27, 2023

Uh oh!

anna-charlotte Feb 27, 2023

Uh oh!

github-actions bot commented Feb 27, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

feat: nested attribute access in find() #1176

feat: nested attribute access in find() #1176

Uh oh!

Conversation

anna-charlotte commented Feb 24, 2023

Uh oh!

Uh oh!

Uh oh!

samsja Feb 27, 2023

Choose a reason for hiding this comment

Uh oh!

anna-charlotte Feb 27, 2023

Choose a reason for hiding this comment

Uh oh!

JohannesMessner Feb 27, 2023

Choose a reason for hiding this comment

Uh oh!

anna-charlotte Feb 27, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 27, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

feat: nested attribute access in `find()` #1176

feat: nested attribute access in `find()` #1176