[storage/journal + storage/adb/any] Make replay ordered #1115
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
b2d28da to
7da98ef
Compare
7da98ef to
abc84d8
Compare
There was a problem hiding this comment.
Pull Request Overview
This PR improves the journal replay functionality by enforcing in-order element production and optimizing performance. Key changes include:
- Removing the replay_concurrency parameter and updating the journal.replay API across modules.
- Adjusting test and configuration setups to align with the new replay API.
- Extending benchmark coverage in a few files by increasing workload sizes.
Reviewed Changes
Copilot reviewed 24 out of 24 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| storage/src/archive/storage.rs | Removed replay_concurrency from journal.replay call |
| storage/src/archive/mod.rs | Removed replay_concurrency from config and tests |
| storage/src/archive/benches/utils.rs | Removed replay_concurrency from benchmark configuration |
| storage/src/adb/any.rs | Updated snapshot building to use new replay API signature |
| runtime/src/utils/buffer/read.rs | Added blob_size() helper method |
| examples/log/src/main.rs | Removed replay_concurrency from example configuration |
| consensus/* and simplex/* files | Removed replay_concurrency from configuration, tests, and actor code |
| consensus/src/ordered_broadcast/* | Removed journal_replay_concurrency from configuration and actor code |
Comments suppressed due to low confidence (2)
storage/src/archive/storage.rs:119
- Ensure the API documentation and in-code comments for journal.replay are updated to reflect the new signature (single replay_buffer parameter) after removing replay_concurrency.
let stream = journal.replay(cfg.replay_buffer).await?;
consensus/src/ordered_broadcast/engine.rs:978
- Update the documentation for journal.replay in this context to clearly indicate that the concurrent replay parameter is no longer used, ensuring consistency with the new API.
.replay(self.journal_replay_buffer)
fdc5cd3 to
9e34f3b
Compare
storage/src/journal/fixed.rs
Outdated
| } | ||
| let start_offset = (start_pos % items_per_blob) * Self::CHUNK_SIZE_U64; | ||
|
|
||
| let stream = stream::iter(blob_plus).flat_map(move |(i, blob_index, blob, sz)| { |
| // Replay all blobs in order and stream items as they are read (to avoid occupying too much | ||
| // memory with buffered data) | ||
| Ok( | ||
| stream::iter(blobs).flat_map( |
a04428f to
4ed1bde
Compare
4ed1bde to
ed2f982
Compare
ed2f982 to
a9f6351
Compare
patrick-ogrady
approved these changes
Jun 17, 2025
Codecov ReportAttention: Patch coverage is
@@ Coverage Diff @@
## main #1115 +/- ##
==========================================
- Coverage 91.03% 91.01% -0.02%
==========================================
Files 191 191
Lines 55316 55231 -85
==========================================
- Hits 50355 50269 -86
- Misses 4961 4962 +1
... and 1 file with indirect coverage changes Continue to review full report in Codecov by Sentry.
🚀 New features to boost your workflow:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Performance on the fixed_replay benchmark is about 25-30% faster.
Archive restart benchmark unaffected.
Addresses #1109.