Update TFDS to 4.2.0 #2912
Merged
Update TFDS to 4.2.0 #2912
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ae3c56a to
c05b831
Compare
API: * Add `tfds build` to the CLI. See [documentation](https://www.tensorflow.org/datasets/cli#tfds_build_download_and_prepare_a_dataset). * DownloadManager now returns [Pathlib-like](https://docs.python.org/3/library/pathlib.html#basic-use) objects * Datasets returned by `tfds.as_numpy` are compatible with `len(ds)` * New `tfds.features.Dataset` to represent nested datasets * Add `tfds.ReadConfig(add_tfds_id=True)` to add a unique identifiant to the example `ex['tfds_id']` (e.g. `b'train.tfrecord-00012-of-01024__123'`) * Add `num_parallel_calls` option to `tfds.ReadConfig` to overwrite to default `AUTOTUNE` option * `tfds.ImageFolder` now support `tfds.decode.SkipDecoder` * Add multichannel audio support to `tfds.features.Audio` * Better `tfds.as_dataframe` visualization (ffmpeg video if installed, bounding boxes,...) * Add `try_gcs` to `tfds.builder(..., try_gcs=True)` * Simpler `BuilderConfig` definition: global `VERSION` and `RELEASE_NOTES` are applied to all `BuilderConfig`. Config description is now optional. Breaking compatibility changes: * Removed non-plain text config of text datasets and remove config: `multi_nli/plain_text` -> `multi_nli` * To guarantee better deterministic, new validations are performed on the keys when creating a dataset (to avoid filenames as keys (non-deterministic) and restrict key to `str`, `bytes` and `int`). New errors likely indicates an issue in the dataset implementation. * `tfds.core.benchmark` now returns a `pd.DataFrame` (instead of a `dict`) * `tfds.units` is not visible anymore from the public API Bug fixes: * Support 0-len sequence with images of dynamic shape (Fix #2616) * Progression bar correctly updated when copying files. * Many bug fixes (GPath consistency with pathlib, s3 compatibility, TQDM visual artifacts, GCS crash on windows, re-download when checksums updated,...) * Better debugging and error message (e.g. human readable size,...) * Allow `max_examples_per_splits=0` in `tfds build --max_examples_per_splits=0` to test `_split_generators` only (without `_generate_examples`). And of course, new datasets and many datasets updates. Thank you the community for their many valuable contributions and to supporting us in this project!!! PiperOrigin-RevId: 350344016
c05b831 to
ccb1bbc
Compare
|
Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). 📝 Please visit https://cla.developers.google.com/ to sign. Once you've signed (or fixed any issues), please reply here with What to do if you already signed the CLAIndividual signers
Corporate signers
ℹ️ Googlers: Go here for more info. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Update TFDS to 4.2.0
API:
tfds buildto the CLI. See documentation.tfds.as_numpyare compatible withlen(ds)tfds.features.Datasetto represent nested datasetstfds.ReadConfig(add_tfds_id=True)to add a unique identifiant to the exampleex['tfds_id'](e.g.b'train.tfrecord-00012-of-01024__123')num_parallel_callsoption totfds.ReadConfigto overwrite to defaultAUTOTUNEoptiontfds.ImageFoldernow supporttfds.decode.SkipDecodertfds.features.Audiotfds.as_dataframevisualization (ffmpeg video if installed, bounding boxes,...)try_gcstotfds.builder(..., try_gcs=True)BuilderConfigdefinition: globalVERSIONandRELEASE_NOTESare applied to allBuilderConfig. Config description is now optional.Breaking compatibility changes:
str,bytesandint). New errors likely indicates an issue in the dataset implementation.tfds.core.benchmarknow returns apd.DataFrame(instead of adict)tfds.unitsis not visible anymore from the public APIBug fixes:
max_examples_per_splits=0intfds build --max_examples_per_splits=0to test_split_generatorsonly (without_generate_examples).