Retrieving doc body in InMemorySource #32
Replies: 2 comments 4 replies
-
|
@kembala thoughts? |
Beta Was this translation helpful? Give feedback.
-
|
Good observation. We think that to build an efficient search system we need to save and transfer the least amount of data during the ingestion and search. The expectation is that on the client side these results who need to be enriched anyway which renders this capability as something that slows down the retrieval. That being said, there could be value in adding this as an optional feature to improve the DX as you pointed out. I have created an issue from this discussion to track this enhancement. |
Beta Was this translation helpful? Give feedback.
-
|
@kembala in combination with chunking, how would it work to not return the actual chunk, just it's ID? would you return an offset into the original doc and it's ID or what? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, one of the options is to return a range in the document. As the chunking is handled internally the chunk ID is not a meaningful artifact for the user, but the ID of the parent document is and the offset is. Apart from latency and storage needs there is nothing that would block adding more context to the results. Meaning that we would be open to extending this functionality, if there is more need. |
Beta Was this translation helpful? Give feedback.
-
|
I think that the need is 100% there to return the whole entity - good place to start, optimize after. |
Beta Was this translation helpful? Give feedback.
-
|
Tracking this at: #33 |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
I know many vec db-s dont have this however - Doc body cant' be retrieved out of the InMemorySource this is important for DX debugging experience, since were going to trouble of building app and making the DX important
Beta Was this translation helpful? Give feedback.
All reactions