Add aggregated format compare #389
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
Pull Request Overview
This PR introduces aggregated format comparison features for benchmark data, with modifications to percentile metrics handling and conversion routines for aggregated input. Key changes include:
- Removal of P25 and P75 percentile comparisons in the benchmark comparison functions.
- Addition of the AsBench converter in ttfb.go and a new LongSeg method in throughput.go.
- Updates in the CLI to support aggregated input and enhanced JSON decoding.
Reviewed Changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| pkg/bench/compare.go | Removed P25/P75 percentile fields from comparison logic |
| pkg/aggregate/ttfb.go | Added AsBench conversion with scaling adjustments |
| pkg/aggregate/throughput.go | Introduced LongSeg for converting segment data |
| pkg/aggregate/compare.go | Added aggregated comparison function |
| cli/cmp.go | Updated CLI input handling for aggregated and legacy files |
| cli/analyze.go | Removed check limiting benchmark files to a single input |
Usage: warp cmp before.jzon.zst after.json.zst`. Example... ``` λ warp cmp warp-mixed-2025-04-22[151603]-BTJN.json.zst warp-mixed-2025-04-22[135713]-6t7p.json.zst ------------------- Operation: DELETE Operations: 812 -> 2232 Duration: 31s -> 1m34s * Average: -8.93% (-2.3) obj/s * Requests: Avg: +31.2ms (+9%), P50: +15.9ms (+12%), P99: +37.6ms (+10%), Best: -12.3ms (-79%), Worst: +58.8ms (+12%) StdDev: +5.5ms (+6%) ------------------- Operation: GET Operations: 3651 -> 10024 Duration: 31s -> 1m34s * Average: -9.00% (-105.3 MiB/s) throughput, -9.00% (-10.5) obj/s * Requests: Avg: +2.5ms (+5%), P50: +900µs (+3%), P99: +1.1ms (+2%), Best: -2.2ms (-22%), Worst: +81.1ms (+51%) StdDev: +1.9ms (+21%) * TTFB: Avg: +1.5ms (+20%), P50: +415.655µs (+7%), P99: +2.007334ms (+4%), Best: -499.8µs (-24%), Worst: +78.9896ms (+54%) StdDev: +2.069452ms (+27%) ------------------- Operation: PUT Operations: 1219 -> 3351 Duration: 31s -> 1m34s * Average: -8.84% (-34.5 MiB/s) throughput, -8.84% (-3.4) obj/s * Requests: Avg: +44.3ms (+13%), P50: +43.2ms (+13%), P99: +65.5ms (+11%), Best: +4.6ms (+7%), Worst: +107.2ms (+16%) StdDev: +13.1ms (+12%) ------------------- Operation: STAT Operations: 2442 -> 6685 Duration: 31s -> 1m34s * Average: -9.31% (-7.3) obj/s * Requests: Avg: +1.3ms (+3%), P50: -100µs (-4%), P99: -200µs (-1%), Best: 0s (0%), Worst: +26.4ms (+35%) StdDev: +100µs (+2%) λ ```
8a151e6 to
1835184
Compare
harshavardhana
approved these changes
Apr 25, 2025
|
Looks like there is a crash here when results are empty |
klauspost
added a commit
to klauspost/warp
that referenced
this pull request
Apr 28, 2025
Fixes crash mentioned on minio#389
Merged
harshavardhana
pushed a commit
that referenced
this pull request
Apr 28, 2025
Fixes crash mentioned on #389
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Usage: warp cmp before.jzon.zst after.json.zst`.
Example...