fastMD5 is a high-performance MD5 checksum tool, designed as a faster and parallelized alternative to the standard md5sum. It offers flexible speed modes, multi-threading support, and efficient directory handling—making it especially well-suited for verifying very large files or entire directories.
- 🚀 Adjustable speed levels (0–9)
- 0 – Full sequential computation, ~20% faster than GNU
md5sum, fully compatible with it. - 1 – Full-file computation with optimized performance, up to > 4× faster than GNU
md5sum. - 2–9 – Block-based sampling for approximate checksums. For example, a file >100 GB can be processed in under 1 second.
- 0 – Full sequential computation, ~20% faster than GNU
- 🧵 Multi-threaded execution – Leverages multi-core CPUs for even faster performance.
- 📂 Recursive directory support – Optionally include hidden files and symbolic links.
- ✅ Checksum verification mode (
--check) – Validate files against precomputed MD5 sums.
# Compute MD5 checksums at the default speed level (5), typically used to verify file integrity after copying.
fastmd5 -t 10 file1.fastq.gz file2.fastq.gz data_directory/
# Run in the most accurate mode (level 0, equivalent to `md5sum`).
fastmd5 -t 3 -s 0 file1.fastq.gz file2.fastq.gz data_directory/
# Run full-file computation with optimized performance (level 1).
fastmd5 -t 5 -s 1 file1.fastq.gz file2.fastq.gz data_directory/
# Verify checksums from a checksum file (equivalent to `md5sum -c`).
fastmd5 --check checksums.md5conda install bioconda::fastmd5fastMD5 is written in rust, try below commands (no root required) or refer here to install Rust first.
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh国内用户请参考这里设置清华源加速
git clone https://github.com/moold/fastMD5.git
cd fastMD5 && cargo build --releasecd test && bash hh.sh# Generate checksum file (default speed level 5)
fastmd5 -t 10 file1.fastq.gz file2.fastq.gz data_directory/ > checksums.md5
# Verify against checksum file
fastmd5 --check checksums.md5# Generate checksum file (speed level 1, full-file computation)
fastmd5 -t 10 -s 1 file1.fastq.gz file2.fastq.gz data_directory/ > checksums.md5
# Verify against checksum file
fastmd5 --check checksums.md5# Generate checksum file (speed level 0, GNU-compatible mode)
fastmd5 -t 10 -s 0 file1.fastq.gz file2.fastq.gz data_directory/ > checksums.md5
# Equivalent GNU command
ls file1.fastq.gz file2.fastq.gz data_directory/ | while read line; do md5sum $line >> checksums.md5; done
# Verify against checksum file
fastmd5 --check checksums.md5
# Equivalent GNU command
md5sum -c checksums.md5ℹ️ Run ./target/release/fastMD5 -h to view all available options.
Feel free to raise an issue at the issue page.
Note: Please ask questions on the issue page first. They are also helpful to other users.
For additional help, please send an email to huj1203_at_foxmail_dot_cn.
| Program | File Size (GB) | Speed Level (-s) | Wall Clock (s) | Speedup (×) |
|---|---|---|---|---|
GNU md5sum |
1 | – | 1.945 | 1.0× |
fastMD5 |
1 | 0 | 1.594 | 1.22× |
fastMD5 |
1 | 1 | 0.454 | 4.29× |
fastMD5 |
1 | 2 | 0.164 | 11.85× |
GNU md5sum |
10 | – | 19.347 | 1.0× |
fastMD5 |
10 | 0 | 15.826 | 1.22× |
fastMD5 |
10 | 1 | 4.387 | 4.41× |
fastMD5 |
10 | 2 | 0.165 | 117.26× |
GNU md5sum |
40 | – | 79.040 | 1.0× |
fastMD5 |
40 | 0 | 63.150 | 1.25× |
fastMD5 |
40 | 1 | 18.738 | 4.22× |
fastMD5 |
40 | 2 | 0.173 | 456.65× |
GNU md5sum |
90 | – | 179.552 | 1.0× |
fastMD5 |
90 | 0 | 141.837 | 1.27× |
fastMD5 |
90 | 1 | 43.649 | 4.11× |
fastMD5 |
90 | 2 | 0.176 | 1020.27× |
Note: Wall-clock times were measured using hyperfine with three warm-up runs on a single CPU core.
You can track updates by tab the Star button on the upper-right corner at the github page.