Skip to content

Conversation

andresilva
Copy link
Collaborator

@andresilva andresilva commented Aug 10, 2025

Updates the variable journal to use the Append blob wrapper for read caching through the buffer pool.
This PR touches a lot of things because of the fallout of having to pass along the PoolRef through multiple components, it can be reviewed commit by commit.

Fixes #1223.

@andresilva
Copy link
Collaborator Author

Let me know if there are any specific tests to the variable journal you think should be added.

@andresilva andresilva force-pushed the andre/variable-journal-append-wrapper branch from dd244e9 to a12c8d1 Compare August 10, 2025 22:58
@andresilva
Copy link
Collaborator Author

Both fixed and variable journals are still using a Read wrapper for replay, I wonder if this is still worth it or whether just using the page cache from Append is now good enough.

@roberto-bayardo roberto-bayardo self-requested a review August 11, 2025 14:15
Copy link
Contributor

@patrick-ogrady patrick-ogrady left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left 2 small nits. Very close IMO.

@patrick-ogrady
Copy link
Contributor

patrick-ogrady commented Aug 11, 2025

Both fixed and variable journals are still using a Read wrapper for replay, I wonder if this is still worth it or whether just using the page cache from Append is now good enough.

I think using Read should still outperform (and will avoid clobbering the cache). We should reserve it for random read workloads.

@roberto-bayardo
Copy link
Collaborator

We can port the any::Fixed benchmark to any::Variable; this PR should result in a very nice performance gain with it.

@roberto-bayardo
Copy link
Collaborator

We can port the any::Fixed benchmark to any::Variable; this PR should result in a very nice performance gain with it.

#1388

@andresilva
Copy link
Collaborator Author

I think all issues have been addressed now.

@roberto-bayardo
Copy link
Collaborator

this looks good to me. let's get the benchmark merged then I'll approve.

@patrick-ogrady
Copy link
Contributor

10x faster!

adb::variable_init/elements=100000 operations=1000000
                        time:   [333.55 ms 338.91 ms 345.30 ms]
                        change: [-89.380% -89.187% -88.948%] (p = 0.00 < 0.05)
                        Performance has improved.
adb::variable_init/elements=100000 operations=2000000
                        time:   [367.78 ms 373.06 ms 378.61 ms]
                        change: [-89.794% -89.615% -89.428%] (p = 0.00 < 0.05)
                        Performance has improved.
adb::variable_init/elements=200000 operations=1000000
                        time:   [614.35 ms 622.39 ms 633.07 ms]
                        change: [-89.272% -89.063% -88.841%] (p = 0.00 < 0.05)
                        Performance has improved.
adb::variable_init/elements=200000 operations=2000000
                        time:   [700.03 ms 722.48 ms 751.12 ms]
                        change: [-89.628% -89.303% -88.869%] (p = 0.00 < 0.05)
                        Performance has improved.

@patrick-ogrady patrick-ogrady merged commit 04363cf into commonwarexyz:main Aug 12, 2025
33 checks passed
Copy link

codecov bot commented Aug 12, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.51%. Comparing base (5ea06c0) to head (b89a71d).
⚠️ Report is 5 commits behind head on main.

@@            Coverage Diff             @@
##             main    #1384      +/-   ##
==========================================
+ Coverage   91.34%   91.51%   +0.17%     
==========================================
  Files         265      265              
  Lines       66726    67134     +408     
==========================================
+ Hits        60950    61441     +491     
+ Misses       5776     5693      -83     
Files with missing lines Coverage Δ
consensus/src/aggregation/engine.rs 84.35% <100.00%> (+0.07%) ⬆️
consensus/src/aggregation/mod.rs 96.63% <100.00%> (+<0.01%) ⬆️
consensus/src/marshal/actor.rs 87.80% <100.00%> (+9.59%) ⬆️
consensus/src/marshal/mod.rs 99.75% <ø> (+0.27%) ⬆️
consensus/src/ordered_broadcast/engine.rs 88.11% <100.00%> (+0.04%) ⬆️
consensus/src/ordered_broadcast/mod.rs 99.56% <100.00%> (+<0.01%) ⬆️
consensus/src/simplex/actors/voter/actor.rs 91.49% <100.00%> (-0.47%) ⬇️
consensus/src/simplex/actors/voter/mod.rs 98.30% <100.00%> (+<0.01%) ⬆️
consensus/src/simplex/config.rs 68.75% <ø> (ø)
consensus/src/simplex/engine.rs 94.56% <100.00%> (+0.05%) ⬆️
... and 17 more

... and 5 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5ea06c0...b89a71d. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[storage/journal/variable] variable::Journal should use BufferPool

3 participants