Skip to content

Improve use of async DocParse via aryn-sdk #1134

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
MarkLindblad merged 21 commits into from
Jan 29, 2025
Merged

Improve use of async DocParse via aryn-sdk #1134

Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
9d24f31
Make error code bounds checking look nicer
MarkLindblad Jan 24, 2025
d6f2ce4
Rename `_set_stream` to `_should_stream`
MarkLindblad Jan 24, 2025
82ff70d
Add comment in docstring about how cancellation can fail if the speci…
MarkLindblad Jan 24, 2025
57b2f1f
Make `partition_file_async_list` example output more concise
MarkLindblad Jan 24, 2025
5b7080a
Refactor shared header code into function, add `User-Agent`
MarkLindblad Jan 27, 2025
f14be07
Remove `test_partition_file_async`
MarkLindblad Jan 27, 2025
2cb94f8
Remove example from `partition_file_async_list` docstring
MarkLindblad Jan 27, 2025
6968d85
Keep all async examples in aryn-sdk README.md, point there from docst…
MarkLindblad Jan 27, 2025
56afd3a
Consolidate url rewriting into one function
MarkLindblad Jan 27, 2025
796c442
Add comment making behavior more obvious
MarkLindblad Jan 27, 2025
5c9e7a9
Remove dependence on `importlib`
MarkLindblad Jan 28, 2025
ec8f348
Make `_convert_sync_to_async_url` use more readable, fix linting
MarkLindblad Jan 28, 2025
ba11fb9
Simplify UX of `partition_file_async_list`
MarkLindblad Jan 28, 2025
80fca40
Filter out non-DocParse jobs from `partition_file_async_list`
MarkLindblad Jan 28, 2025
4a1d50c
Make `test_multiple_partition_file_async` more robust
MarkLindblad Jan 28, 2025
a49f7e9
Make `_convert_sync_to_async_url`'s `truncate` a keyword argument
MarkLindblad Jan 29, 2025
0f1cdb0
Improve performance of `partition_file_async_list`
MarkLindblad Jan 29, 2025
a7b7c44
Fix return description in docstring for `partition_file_async_result`
MarkLindblad Jan 29, 2025
a69af61
Improve return type in docstring of `partition_file_async_result`
MarkLindblad Jan 29, 2025
c069558
Remove repetitive sentence in docstring for `partition_file_async_res…
MarkLindblad Jan 29, 2025
7fba9f0
Fix `aryn-sdk` notebook example `ArynPartitionerPython.ipynb`
MarkLindblad Jan 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Filter out non-DocParse jobs from partition_file_async_list
  • Loading branch information
@MarkLindblad
MarkLindblad committed Jan 28, 2025
commit 80fca403c4037c10fb5d89b31a926c445853e258
10 changes: 6 additions & 4 deletions lib/aryn-sdk/aryn_sdk/partition/partition.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -524,10 +524,12 @@ def partition_file_async_list(
headers = _generate_headers(aryn_config.api_key())
response = requests.get(async_list_url, headers=headers, stream=_should_stream(), verify=ssl_verify)

result = response.json()
result = result["jobs"]
for job_id in result.keys():
del result[job_id]["path"]
all_jobs = response.json()["jobs"]
result = {}
for job_id in all_jobs.keys():
if all_jobs[job_id]["path"] == "/v1/document/partition":
result[job_id] = all_jobs[job_id]
del all_jobs[job_id]["path"]
return result


Expand Down
Loading