Commits

Branch selector

 main

User selector

All users

Datepicker

All time

Commit History

Commits on Sep 24, 2025

• Update grain version in airio

  Show description for 4e3b376
  

  gauravmishra

  authored and

  AirIO

  committed
  4e3b376

  Copy full SHA for 4e3b376

Commits on May 7, 2025

• Simplify state of `MapDatasetIterator`.

  Show description for 7feb9dd
  

  iindyk

  authored and

  AirIO

  committed
  7feb9dd

  Copy full SHA for 7feb9dd

Commits on Apr 28, 2025

• No public description

  Show description for debc14b
  

  h-joo

  authored and

  AirIO

  committed
  debc14b

  Copy full SHA for debc14b

Commits on Apr 23, 2025

• Fix AirIO hashing in Mixtures. hash() is not stable across processes.

  Show description for 167cd61
  

  timshen91

  authored and

  AirIO

  committed
  167cd61

  Copy full SHA for 167cd61

Commits on Apr 18, 2025

• Validate that all leaf tasks in AirIO mixtures are unique and update the ordering of `leaf_tasks` based on insertion order rather than task name.

  Show description for e6451bf
  

  AirIO Team

  authored and

  AirIO

  committed
  e6451bf

  Copy full SHA for e6451bf

• Fix airio benchmark tests.

  Show description for 059560a
  

  gauravmishra

  authored and

  AirIO

  committed
  059560a

  Copy full SHA for 059560a

Commits on Apr 10, 2025

• Configure tests to explicitly use jax_threefry_partitionable=False.

  Show description for 751e506
  

  jburnim

  authored and

  AirIO

  committed
  751e506

  Copy full SHA for 751e506

Commits on Mar 4, 2025

• Migrate `tf.data.experimental.RandomDataset` to `tf.data.Dataset.random`.

  Show description for c6531b3
  

  yangustc07

  authored and

  AirIO

  committed
  c6531b3

  Copy full SHA for c6531b3

Commits on Oct 18, 2024

• Introduces DatasetIterator._parents and DatasetIterator._parent.

  Show description for 37109ff
  

  iindyk

  authored and

  AirIO

  committed
  37109ff

  Copy full SHA for 37109ff

Commits on Aug 19, 2024

• Improve error propagation.

  Show description for d08752e
  

  iindyk

  authored and

  AirIO

  committed
  d08752e

  Copy full SHA for d08752e

Commits on Aug 6, 2024

• Switch to graduated grain.Dataset APIs.

  Show description for 815f40e
  

  iindyk

  authored and

  AirIO

  committed
  815f40e

  Copy full SHA for 815f40e

Commits on Jul 3, 2024

• Publicly expose UnigramVocabulary

  Show description for 994e213
  

  AirIO Team

  authored and

  AirIO

  committed
  994e213

  Copy full SHA for 994e213

Commits on Jul 2, 2024

• No public description

  Show description for a456edd
  

  AirIO Team

  authored and

  AirIO

  committed
  a456edd

  Copy full SHA for a456edd

Commits on Jun 28, 2024

• Add batching support to t5 denoise preprocessor.

  Show description for cbeb815
  

  gauravmishra

  authored and

  AirIO

  committed
  cbeb815

  Copy full SHA for cbeb815

Commits on Jun 20, 2024

• Fix `ataset_providers_test.py`.

  Show description for 2c48ef0
  

  AirIO Team

  authored and

  AirIO

  committed
  2c48ef0

  Copy full SHA for 2c48ef0

Commits on Jun 18, 2024

• Improve batching error messages.

  Show description for e7620ba
  

  Marvin182

  authored and

  AirIO

  committed
  e7620ba

  Copy full SHA for e7620ba

Commits on Jun 13, 2024

• Add a UnigramVocabulary

  Show description for 66721d6
  

  AirIO Team

  authored and

  AirIO

  committed
  66721d6

  Copy full SHA for 66721d6

Commits on May 16, 2024

• Replace deprecated `jax.tree_*` functions with `jax.tree.*`

  Show description for 2582307
  

  Jake VanderPlas

  authored and

  AirIO

  committed
  2582307

  Copy full SHA for 2582307

Commits on Apr 24, 2024

• Fix infinite repeat for GrainTask.get_lazy_datasets.

  Show description for d96858e
  

  gauravmishra

  authored and

  AirIO

  committed
  d96858e

  Copy full SHA for d96858e

Commits on Apr 22, 2024

• Fix a bug in Task.get_dataset() when using lazy_dataset and preprocessors that have update_runtime_args.

  Show description for 36a6650
  

  gauravmishra

  authored and

  AirIO

  committed
  36a6650

  Copy full SHA for 36a6650

• Add eos support in airio-pygrain tokenizer.

  Show description for 3af8663
  

  gauravmishra

  authored and

  AirIO

  committed
  3af8663

  Copy full SHA for 3af8663

Commits on Apr 19, 2024

• Fix post-mix preprocessing for AirIO-PyGrain Mixtures.

  Show description for f0fe9e0
  

  gauravmishra

  authored and

  AirIO

  committed
  f0fe9e0

  Copy full SHA for f0fe9e0

Commits on Apr 18, 2024

• Convert tensors to numpy at the end of span_corruption preprocessor in AirIO-PyGrain

  Show description for 133d00a
  

  gauravmishra

  authored and

  AirIO

  committed
  133d00a

  Copy full SHA for 133d00a

Commits on Apr 17, 2024

• internal

  Show description for 7401d08
  

  gauravmishra

  authored and

  AirIO

  committed
  7401d08

  Copy full SHA for 7401d08

Commits on Apr 16, 2024

• Add tests to load and parse tf.Examples from files with airio pygrain datasources.

  Show description for a296623
  

  gauravmishra

  authored and

  AirIO

  committed
  a296623

  Copy full SHA for a296623

Commits on Apr 15, 2024

• remove with_eos from Tokenizer class.

  Show description for 0cf7619
  

  gauravmishra

  authored and

  AirIO

  committed
  0cf7619

  Copy full SHA for 0cf7619

Commits on Apr 11, 2024

• Update test data to use tf.train.Example protos.

  Show description for dee4d99
  

  gauravmishra

  authored and

  AirIO

  committed
  dee4d99

  Copy full SHA for dee4d99

Commits on Apr 9, 2024

• Remove extra features before packing.

  Show description for 2f93084
  

  gauravmishra

  authored and

  AirIO

  committed
  2f93084

  Copy full SHA for 2f93084

Commits on Apr 5, 2024

• Merge pull request #226 from gspschmid/gschmid/bump-grain-007

  Show description for cfca4a1
  texasmichelle

  authored
  cfca4a1

  Copy full SHA for cfca4a1

• Merge branch 'main' into gschmid/bump-grain-007

  texasmichelle

  authored
  267efd5

  Copy full SHA for 267efd5

• Add json benchmarks for pygrain.

  Show description for 9180492
  

  texasmichelle

  authored and

  AirIO

  committed
  9180492

  Copy full SHA for 9180492

• Add arrayrecord benchmarks for pygrain.

  Show description for d5a76ff
  

  texasmichelle

  authored and

  AirIO

  committed
  d5a76ff

  Copy full SHA for d5a76ff

• Clean up pygrain data source benchmarks.

  Show description for d8794ab
  

  texasmichelle

  authored and

  AirIO

  committed
  d8794ab

  Copy full SHA for d8794ab

• Add batch_size to AirIOInjectedRuntimeArgs.

  Show description for 55a4ade
  

  gauravmishra

  authored and

  AirIO

  committed
  55a4ade

  Copy full SHA for 55a4ade

• Clean up pygrain benchmarks.

  Show description for 71ad7c0
  

  texasmichelle

  authored and

  AirIO

  committed
  71ad7c0

  Copy full SHA for 71ad7c0

Pagination

Previous

Next