[coding] Introduce commonware-coding
#1272
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
There was a problem hiding this comment.
Pull Request Overview
Introduce the commonware-coding crate for erasure-coded data recovery using Reed–Solomon with Merkle proofs.
- Add the new
commonware-codingcrate, including implementation, docs, tests, and benchmarks. - Update workspace manifests and top-level documentation to include the
codingpackage. - Integrate CI workflows for benchmarking and publishing the new crate.
Reviewed Changes
Copilot reviewed 12 out of 13 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/index.html | Add link to new commonware-coding crate in documentation |
| coding/src/reed_solomon/mod.rs | Implement Reed–Solomon encode/decode with BMT proofs |
| coding/src/reed_solomon/benches/encode.rs | Add Criterion benchmark for encode |
| coding/src/reed_solomon/benches/decode.rs | Add Criterion benchmark for decode |
| coding/src/reed_solomon/benches/bench.rs | Define benchmark entrypoint |
| coding/src/lib.rs | Declare reed_solomon module |
| coding/README.md | Add crate README |
| coding/Cargo.toml | Add crate manifest for commonware-coding |
| README.md | Include coding in top-level README |
| Cargo.toml | Add coding to workspace members |
| .github/workflows/slow.yml | Add benchmarking job for commonware-coding |
| .github/workflows/publish.yml | Add publish step for the coding crate |
Comments suppressed due to low confidence (2)
coding/src/reed_solomon/benches/decode.rs:27
- [nitpick] The variable
proofsactually holds the vector ofChunks returned byencode. Consider renaming it tochunksfor clarity.
let (root, proofs) = encode::<Sha256>(chunks, min, data).unwrap();
coding/src/reed_solomon/mod.rs:128
- [nitpick] The error message
"inconsistent"is vague. It may be clearer to specify that the recomputed Merkle root did not match the provided root (e.g.,"inconsistent root: reconstructed data does not match").
#[error("inconsistent")]
This was referenced Jul 14, 2025
Codecov ReportAttention: Patch coverage is
@@ Coverage Diff @@
## main #1272 +/- ##
==========================================
+ Coverage 91.04% 91.06% +0.02%
==========================================
Files 245 246 +1
Lines 59760 60120 +360
==========================================
+ Hits 54407 54748 +341
- Misses 5353 5372 +19
Continue to review full report in Codecov by Sentry.
🚀 New features to boost your workflow:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related: #1267
To ensure all participants see a notarized block in consensus, the leader should send chunks to each participant. If a chunk is valid (proof verifies), participants should multicast their chunk when sending their vote. When any honest node sees 2f+1 votes, they'll be able to recover the underlying block (set to only require f+1 chunks to derive) or all will determine that the encoding was invalid (and skip it). This approach is a much more explicit defense against what is usually not considered to be a problem with the "gossip model of block dissemination".
Conveniently, this technique can also increase throughput by making broadcast of transaction data "fair"/"symmetric" (rather than relying entirely on the leader's egress capacity) for negligible latency overhead.
Related
Performance
reed_solomon::encode/msg_len=256 chunks=10 time: [8.8251 µs 8.8957 µs 9.0693 µs] reed_solomon::encode/msg_len=256 chunks=25 time: [17.640 µs 17.742 µs 17.854 µs] reed_solomon::encode/msg_len=256 chunks=50 time: [33.731 µs 33.937 µs 34.241 µs] reed_solomon::encode/msg_len=256 chunks=100 time: [67.393 µs 69.226 µs 70.532 µs] reed_solomon::encode/msg_len=256 chunks=250 time: [165.40 µs 166.97 µs 168.45 µs] reed_solomon::encode/msg_len=4096 chunks=10 time: [38.725 µs 40.569 µs 41.853 µs] reed_solomon::encode/msg_len=4096 chunks=25 time: [47.959 µs 48.297 µs 48.916 µs] reed_solomon::encode/msg_len=4096 chunks=50 time: [64.271 µs 64.364 µs 64.520 µs] reed_solomon::encode/msg_len=4096 chunks=100 time: [97.587 µs 98.226 µs 99.510 µs] reed_solomon::encode/msg_len=4096 chunks=250 time: [166.86 µs 168.42 µs 172.05 µs] reed_solomon::encode/msg_len=65536 chunks=10 time: [522.77 µs 525.98 µs 528.30 µs] reed_solomon::encode/msg_len=65536 chunks=25 time: [509.96 µs 512.39 µs 514.03 µs] reed_solomon::encode/msg_len=65536 chunks=50 time: [532.76 µs 534.99 µs 537.79 µs] reed_solomon::encode/msg_len=65536 chunks=100 time: [568.21 µs 570.41 µs 572.14 µs] reed_solomon::encode/msg_len=65536 chunks=250 time: [648.49 µs 651.05 µs 652.77 µs] reed_solomon::encode/msg_len=524288 chunks=10 time: [4.1527 ms 4.1782 ms 4.2223 ms] reed_solomon::encode/msg_len=524288 chunks=25 time: [4.0145 ms 4.0431 ms 4.0703 ms] reed_solomon::encode/msg_len=524288 chunks=50 time: [4.0935 ms 4.1613 ms 4.2308 ms] reed_solomon::encode/msg_len=524288 chunks=100 time: [4.0770 ms 4.0834 ms 4.0980 ms] reed_solomon::encode/msg_len=524288 chunks=250 time: [4.1464 ms 4.2080 ms 4.3002 ms] reed_solomon::encode/msg_len=1048576 chunks=10 time: [8.3454 ms 8.4961 ms 8.6545 ms] reed_solomon::encode/msg_len=1048576 chunks=25 time: [8.0066 ms 8.1024 ms 8.2239 ms] reed_solomon::encode/msg_len=1048576 chunks=50 time: [8.2229 ms 8.3687 ms 8.4814 ms] reed_solomon::encode/msg_len=1048576 chunks=100 time: [8.1020 ms 8.1640 ms 8.2396 ms] reed_solomon::encode/msg_len=1048576 chunks=250 time: [8.1452 ms 8.2088 ms 8.3528 ms] reed_solomon::encode/msg_len=16777216 chunks=10 time: [135.32 ms 136.62 ms 138.16 ms] reed_solomon::encode/msg_len=16777216 chunks=25 time: [131.20 ms 134.92 ms 141.61 ms] reed_solomon::encode/msg_len=16777216 chunks=50 time: [130.96 ms 132.83 ms 134.83 ms] reed_solomon::encode/msg_len=16777216 chunks=100 time: [133.15 ms 133.77 ms 134.47 ms] reed_solomon::encode/msg_len=16777216 chunks=250 time: [135.05 ms 140.31 ms 149.77 ms] reed_solomon::decode/msg_len=256 chunks=10 time: [535.91 µs 552.17 µs 575.03 µs] reed_solomon::decode/msg_len=256 chunks=25 time: [555.53 µs 587.77 µs 643.32 µs] reed_solomon::decode/msg_len=256 chunks=50 time: [601.87 µs 614.26 µs 633.59 µs] reed_solomon::decode/msg_len=256 chunks=100 time: [681.04 µs 715.38 µs 770.70 µs] reed_solomon::decode/msg_len=256 chunks=250 time: [919.56 µs 925.08 µs 932.09 µs] reed_solomon::decode/msg_len=4096 chunks=10 time: [571.41 µs 575.03 µs 577.42 µs] reed_solomon::decode/msg_len=4096 chunks=25 time: [596.11 µs 598.91 µs 602.15 µs] reed_solomon::decode/msg_len=4096 chunks=50 time: [632.51 µs 643.29 µs 675.90 µs] reed_solomon::decode/msg_len=4096 chunks=100 time: [710.67 µs 712.65 µs 714.09 µs] reed_solomon::decode/msg_len=4096 chunks=250 time: [908.16 µs 911.05 µs 915.27 µs] reed_solomon::decode/msg_len=65536 chunks=10 time: [1.2653 ms 1.2751 ms 1.2862 ms] reed_solomon::decode/msg_len=65536 chunks=25 time: [1.2961 ms 1.3089 ms 1.3169 ms] reed_solomon::decode/msg_len=65536 chunks=50 time: [1.3124 ms 1.3333 ms 1.3483 ms] reed_solomon::decode/msg_len=65536 chunks=100 time: [1.4803 ms 1.5452 ms 1.6347 ms] reed_solomon::decode/msg_len=65536 chunks=250 time: [1.6853 ms 1.6919 ms 1.6984 ms] reed_solomon::decode/msg_len=524288 chunks=10 time: [6.8521 ms 6.9267 ms 7.0047 ms] reed_solomon::decode/msg_len=524288 chunks=25 time: [6.7535 ms 6.7918 ms 6.8342 ms] reed_solomon::decode/msg_len=524288 chunks=50 time: [6.9652 ms 7.0132 ms 7.0613 ms] reed_solomon::decode/msg_len=524288 chunks=100 time: [7.3252 ms 7.3541 ms 7.3777 ms] reed_solomon::decode/msg_len=524288 chunks=250 time: [7.6919 ms 7.8995 ms 8.1893 ms] reed_solomon::decode/msg_len=1048576 chunks=10 time: [13.495 ms 13.650 ms 13.752 ms] reed_solomon::decode/msg_len=1048576 chunks=25 time: [13.000 ms 13.120 ms 13.247 ms] reed_solomon::decode/msg_len=1048576 chunks=50 time: [13.455 ms 13.555 ms 13.619 ms] reed_solomon::decode/msg_len=1048576 chunks=100 time: [14.586 ms 15.857 ms 16.941 ms] reed_solomon::decode/msg_len=1048576 chunks=250 time: [14.654 ms 15.304 ms 15.959 ms] reed_solomon::decode/msg_len=16777216 chunks=10 time: [209.70 ms 214.36 ms 217.93 ms] reed_solomon::decode/msg_len=16777216 chunks=25 time: [209.20 ms 211.23 ms 213.92 ms] reed_solomon::decode/msg_len=16777216 chunks=50 time: [213.72 ms 214.74 ms 215.81 ms] reed_solomon::decode/msg_len=16777216 chunks=100 time: [242.13 ms 246.55 ms 252.06 ms] reed_solomon::decode/msg_len=16777216 chunks=250 time: [234.93 ms 243.60 ms 259.74 ms]