What's Changed

• FIM script modifications by @regan-huff in #258
• Adding tool to reshard npy files based on maximum desired size. by @soldni in #269

New Contributors

• @regan-huff made their first contribution in #258

Full Changelog: v1.2.0...v1.2.1

What's Changed

• Code-prose-composition tagger. by @no0p in #247
• Support for WARC resource record types. by @no0p in #248
• Bump artifacts version to 4.4.1 by @no0p in #252
• Sanitize-concat-fim by @cmwilhelm in #253
• Use original s3 path to delete local cache by @CodeCreator in #257
• Safe tokenization by skipping failing docs. by @soldni in #245
• Update RedPajama branch link by @guspan-tanadi in #263
• Skipping empty tagger key instead of erroring out by @Whattabatt in #262
• Tokenizer over custom fields and w/o IDs; BOS/EOS tokens. by @soldni in #266

New Contributors

• @no0p made their first contribution in #247
• @CodeCreator made their first contribution in #257
• @guspan-tanadi made their first contribution in #263

Full Changelog: v1.1.2...v1.2.0

What's Changed

• Matrix targets stomping each other by @undfined in #236
• Artifact name change in release too by @undfined in #237
• Pattern match for all artifacts by @undfined in #239
• Handle retries in the aws client, use adaptive backoff by @undfined in #241
• Bump version to 1.1.2 for release by @undfined in #243

Full Changelog: v1.1.1...v1.1.2

What's Changed

• Fixes checks by @cmwilhelm in #232
• Adds strict filter mechanism to Mixer by @undfined in #231
• v3 download-artifact GH action deprecated by @undfined in #235

Full Changelog: v1.1.0...v1.1.1

What's Changed

• Bump version with postfix for PyPI by @undfined in #206
• Pin maturin in CI by @undfined in #207
• Also pin maturin in action by @undfined in #208
• 'File partition' option and 'document' directory specification by @Whattabatt in #213
• Fixed issues and improved documentation in getting-started.md by @aman-17 in #216
• Mixer validator by @mariia-iureva in #215
• Typo in optional dependency (lingua) check by @soldni in #221

New Contributors

• @aman-17 made their first contribution in #216
• @mariia-iureva made their first contribution in #215

Full Changelog: v1.0.14...v1.0.15

What's Changed

• Bump version with postfix for PyPI by @undfined in #206
• Pin maturin in CI by @undfined in #207
• Also pin maturin in action by @undfined in #208

Full Changelog: v1.0.14...v1.0.14.post1

What's Changed

• Adds dclm fasttext classifier by @undfined in #205

Full Changelog: v1.0.13...v1.0.14

What's Changed

• Bump actions/download-artifact from 3 to 4.1.7 in /.github/workflows in the github_actions group across 1 directory by @dependabot in #198
• Fix bug in length filtering for deduping by @soldni in #197
• Polymorphic span replacement by @undfined in #200
• Dependabot fail, match upload/download action versions by @undfined in #202
• [Json fooramt error in line 133] Update getting-started.md by @yushengsu-thu in #196
• Revert upload/download to v3 for now by @undfined in #203
• Undfined/runner v3 by @undfined in #204

Full Changelog: v1.0.12...v1.0.13

What's Changed

• Added tokenizers for length by @soldni in #189
• Update getting-started.md by @yushengsu-thu in #193
• Bump nltk from 3.8.1 to 3.9 in the pip group across 1 directory by @dependabot in #187
• Use Numpy v1.x instead of 2.x by @soldni in #195

New Contributors

• @yushengsu-thu made their first contribution in #193

Full Changelog: v1.0.11...v1.0.12

What's Changed

• Changed entrypoint to increase IntelliJ compatibility by @soldni in #188

Full Changelog: v1.0.10...v1.0.11