An end-to-end tool for curating GitHub repositories into structured code datasets.

• Fast parallel processing - Download and extract with configurable workers
• Smart filtering - Only processes programming files using GitHub Linguist
• GPT-2 tokenization - Ready-to-use token counts for ML workflows
• Efficient caching - Uses ETags to avoid re-downloading unchanged repos

Perfect for curating training data, running code analysis, or creating repository archives.

cargo install --path .

Create an input file with one GitHub repository per line:

"microsoft/vscode"
"vercel/next.js"
"tensorflow/tensorflow"
"bitcoin/bitcoin"
"rust-lang/rust"
"kubernetes/kubernetes"
"facebook/react"
"docker/compose"
"ansible/ansible"
"elastic/elasticsearch"

Download repositories:

codecurator download ./configs/repos.jsonl

This creates ZIP files in /zip/ directory. Downloads from main branch first, falls back to master if needed.

Extract and process:

codecurator extract ./configs/repos.jsonl --languages Python Rust Verilog

Processes all programming files, tokenizes content, and outputs structured data to /jsonl/ directory.

Deduplication:

codecurator dedupe ./configs/repos.jsonl

Hashes the contents of all files and deduplicates them. Stores the final data to /dedup/ by default.

Statistics:

$ bash stats/count_records.sh ./jsonl/
Total records: 110645

$ bash stats/count_tokens.sh ./dedup/
Total tokens: 346574283