Fix anonymous reading in materialize and add rate limited logging. #898
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
eric-anderson
commented
Oct 10, 2024
- In materialize, try reading using the credentials, but if it doesn't work, fall back to reading anonymously if that seems to be working.
- Add rate limited logging to reading via materialize in local mode.
- Check for no root before checking if a source since that makes more sense.
- switch ntsb_loader_materialized.py over to read in local mode, it was working (with the anonymous fix), but was very slow hence the logging.
* In materialize, try reading using the credentials, but if it doesn't work, fall back to reading anonymously if that seems to be working. * Add rate limited logging to reading via materialize in local mode. * Check for no root before checking if a source since that makes more sense. * switch ntsb_loader_materialized.py over to read in local mode, it was working (with the anonymous fix), but was very slow hence the logging.
bsowell
approved these changes
Oct 10, 2024
Contributor
bsowell
left a comment
There was a problem hiding this comment.
Couple of minor questions, but otherwise LGTM.
Comment on lines
-101
to
+132
| if self._will_be_source(): | ||
| if self._root is None: |
Contributor
There was a problem hiding this comment.
Is there any meaning to this ordering change, or is it just to alphabetize?
Collaborator
Author
There was a problem hiding this comment.
Check for no root before checking if a source since that makes more sense.
Comment on lines
+239
to
+240
| limited_logger = logging.getLogger(__name__ + ".limited_local_source") | ||
| limited_logger.addFilter(LoggerFilter()) |
Contributor
There was a problem hiding this comment.
Is there a reason you define limited_logger at the top of the file and then redefine it here? Doesn't seem like you use it anywhere other than this method.
Collaborator
Author
There was a problem hiding this comment.
Oops, it was only supposed to be down below. You need to have a separate limited logger for each thing you want to be separately rate limited.
dtecuci
pushed a commit
that referenced
this pull request
Oct 14, 2024
) * Fix anonymous reading in materialize and add rate limited logging. * In materialize, try reading using the credentials, but if it doesn't work, fall back to reading anonymously if that seems to be working. * Add rate limited logging to reading via materialize in local mode. * Check for no root before checking if a source since that makes more sense. * switch ntsb_loader_materialized.py over to read in local mode, it was working (with the anonymous fix), but was very slow hence the logging.
dtecuci
added a commit
that referenced
this pull request
Oct 14, 2024
* added ability to read schema from file * small typo Co-authored-by: Matt Welsh <[email protected]> * fixed two funtion refs that were modified * reformatted file with black * fixed schema file format (was json), added more exception handling * Fix anonymous reading in materialize and add rate limited logging. (#898) * Fix anonymous reading in materialize and add rate limited logging. * In materialize, try reading using the credentials, but if it doesn't work, fall back to reading anonymously if that seems to be working. * Add rate limited logging to reading via materialize in local mode. * Check for no root before checking if a source since that makes more sense. * switch ntsb_loader_materialized.py over to read in local mode, it was working (with the anonymous fix), but was very slow hence the logging. * Bump version to v0.1.23. (#903) * fix asdict in the reader too. duh (#907) Signed-off-by: Henry Lindeman <[email protected]> * Add text reprentation for empty tables (#909) * Refactor logical plan serialization. (#905) * Working on this. * Working on refactoring. * Tests pass - is such a thing even possible? * Fix tests. * Fix mypy. * Cleanup. * Fix NTSB examples. * A few tweaks to the query planner prompt, and a workaround in queryui/util.py. * Fix mypy. * seriously small performance improvement that matters when youre processing tens of thousands of tables (from training code) (#906) Signed-off-by: Henry Lindeman <[email protected]> * Handle opensearch reader doc resconstruction when no parent doc in results (#908) * Fix bug in entity extraction. (#911) * Notebooks like default-prep-script.ipynb would fail because the wrong way of generating the prompt would be used. * Rename test to match with name of file being tested. * Fix existing tests to verify parameters on all branches -- the reason the tests were passing was that it was taking the default branch in the test cases * Update all of the tests to directly call run rather than route everything through ray. * Enable copying of the hash context. (#910) * Enable copying of the hash context. * Address comments. * Add option to extract line-based bounding boxes from pdfminer. (#874) We have been using pdfminer's layout detection to group text into boxes. This can cause issues, especially with table extraction, when the boxes don't line up with cells or what we detect with the DETR model. This change adds support for an object_type parameter to the PdfMinerExtractor that can be set to "boxes" (the current behavior), or "lines", which groups characters into lines, but does not group them further. To avoid an explosion of options, we introduce a "text_extractor_options" dict as a paramter, and refactor the TextExtractor class hierarchy a bit to support it. * Support random sample in local mode. (#913) This transform isn't widely used, but still worth supporting in local model to bring it to parity. * Opensearch kwargs fix (#914) * Fix kwargs in opensearch reader * simplify test assertion * lint * pr comments * fix typo (#917) * Update using_jupyter.md (#902) * Update using_jupyter.md Update link * Fixed path --------- Co-authored-by: dtecuci <[email protected]> * Rebased. Added ability to read schema from file * rebased. small typo Co-authored-by: Matt Welsh <[email protected]> * rebased. reformatted file with black * resolved conflicts * changed schema file format to yaml * removed unused import * small typos fixed * fixed spacing --------- Signed-off-by: Henry Lindeman <[email protected]> Co-authored-by: Matt Welsh <[email protected]> Co-authored-by: Eric Anderson <[email protected]> Co-authored-by: Ben Sowell <[email protected]> Co-authored-by: Henry Lindeman <[email protected]> Co-authored-by: Dhruv Kaliraman <[email protected]> Co-authored-by: Vinayak Thapliyal <[email protected]> Co-authored-by: Alex Meyer <[email protected]> Co-authored-by: Karan Sampath <[email protected]> Co-authored-by: jonfritz <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.