Skip to content

Retain element doc_id from source #1178

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
austintlee merged 7 commits into from
Feb 14, 2025
Merged

Retain element doc_id from source #1178

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
8993289
Add a client for Aryn, use the new client to read docs
austin-aryn-ai Feb 8, 2025
d2c7ea7
Use list_docs and get_doc for reading from Aryn
austin-aryn-ai Feb 11, 2025
0150f7f
Mark Aryn classes experimental
austin-aryn-ai Feb 11, 2025
f5fa36a
Add support for docset update
austin-aryn-ai Feb 12, 2025
570b08c
Track element's doc_ids when reading from Aryn
austin-aryn-ai Feb 12, 2025
fe6d778
Make num_of_elements optional in LLMProprtyExtractor
austin-aryn-ai Feb 12, 2025
1d6e644
Fix lint
austin-aryn-ai Feb 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Make num_of_elements optional in LLMProprtyExtractor
  • Loading branch information
@austin-aryn-ai
austin-aryn-ai committed Feb 14, 2025
commit fe6d7783cabb9d2fb6de876672b07e5379e883b4
5 changes: 3 additions & 2 deletions lib/sycamore/sycamore/transforms/extract_schema.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ def __init__(
llm: LLM,
schema_name: Optional[str] = None,
schema: Optional[Union[dict[str, str], Schema]] = None,
num_of_elements: int = 10,
num_of_elements: Optional[int] = None,
prompt_formatter: Callable[[list[Element]], str] = element_list_formatter,
):
super().__init__()
Expand Down Expand Up @@ -221,7 +221,8 @@ def as_llm_map(self, child: Optional[Node], **kwargs) -> Node:

if self._schema_name is not None:
prompt = prompt.set(entity=self._schema_name)
prompt = prompt.set(num_elements=self._num_of_elements)
if self._num_of_elements is not None:
prompt = prompt.set(num_elements=self._num_of_elements)
if self._prompt_formatter is not element_list_formatter:
prompt = prompt.set(prompt_formatter=self._prompt_formatter)

Expand Down