feat(ss-bench): support calibrating metric parameter #740

Sunt-ing · 2022-03-08T04:07:13Z

What's changed and what's your intention?

By comparing performance calculated by the metric system and ss-bench itself, we can adjust the bucket parameters (such as bucket number and size) of the state store metric system.

Checklist

I have written necessary docs and comments
I have added necessary unit tests and integration tests

Refer to a related PR or issue link (optional)

codecov · 2022-03-08T04:23:06Z

Codecov Report

Merging #740 (4e269a8) into main (33d427e) will decrease coverage by 0.05%.
The diff coverage is 58.27%.

@@             Coverage Diff              @@
##               main     #740      +/-   ##
============================================
- Coverage     72.33%   72.27%   -0.06%     
  Complexity     2758     2758              
============================================
  Files           907      908       +1     
  Lines         52466    52520      +54     
  Branches       1783     1783              
============================================
+ Hits          37949    37959      +10     
- Misses        13625    13669      +44     
  Partials        892      892

Flag	Coverage Δ
java	`61.16% <ø> (ø)`
rust	`77.02% <58.27%> (-0.09%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
rust/bench/ss_bench/main.rs	`0.00% <ø> (ø)`
rust/bench/ss_bench/operations/mod.rs	`0.00% <0.00%> (ø)`
rust/bench/ss_bench/utils/display_stat.rs	`0.00% <0.00%> (ø)`
rust/bench/ss_bench/utils/my_metrics.rs	`83.80% <83.80%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 33d427e...4e269a8. Read the comment docs.

Sunt-ing · 2022-03-08T05:53:05Z

Output example:

~/code/risingwave-dev/rust   sunt_move_to_metric_system2  cargo run --bin ss-bench -- --benchmarks "writebatch,getrandom,getseq,prefixscanrandom,deleterandom" --batch-size 1000 --reads 1000 --writes 10000 --deletes 1000 --scans 1000 --statistics --concurrency-num 4 --calibrate-metric
   Compiling risingwave-pb v0.1.0 (/Users/sunt/code/risingwave-dev/rust/prost)
   Compiling risingwave_common v0.1.3 (/Users/sunt/code/risingwave-dev/rust/common)
   Compiling risingwave_rpc_client v0.1.3 (/Users/sunt/code/risingwave-dev/rust/rpc_client)
   Compiling risingwave_storage v0.1.3 (/Users/sunt/code/risingwave-dev/rust/storage)
   Compiling risingwave_bench v0.1.0 (/Users/sunt/code/risingwave-dev/rust/bench)
    Finished dev [unoptimized + debuginfo] target(s) in 20.24s
     Running `target/debug/ss-bench --benchmarks writebatch,getrandom,getseq,prefixscanrandom,deleterandom --batch-size 1000 --reads 1000 --writes 10000 --deletes 1000 --scans 1000 --statistics --concurrency-num 4 --calibrate-metric`
Configurations before preprocess:
 Opts { store: "in-memory", table_size_mb: 256, block_size_kb: 64, bloom_false_positive: 0.1, checksum_algo: "crc32c", benchmarks: "writebatch,getrandom,getseq,prefixscanrandom,deleterandom", concurrency_num: 4, num: 1000000, deletes: 1000, reads: 1000, scans: 1000, writes: 10000, batch_size: 1000, key_size: 16, key_prefix_size: 5, keys_per_prefix: 10, value_size: 100, seed: 0, statistics: true, calibrate_metric: true }
Configurations after preprocess:
 Opts { store: "in-memory", table_size_mb: 256, block_size_kb: 64, bloom_false_positive: 0.1, checksum_algo: "crc32c", benchmarks: "writebatch,getrandom,getseq,prefixscanrandom,deleterandom", concurrency_num: 4, num: 1000000, deletes: 1000, reads: 1000, scans: 1000, writes: 10000, batch_size: 1000, key_size: 16, key_prefix_size: 5, keys_per_prefix: 10, value_size: 100, seed: 0, statistics: true, calibrate_metric: true }

    writebatch
      latency:
        min: 3.120 milli sec,
        mean: 8.904 milli sec,
        p50: 10.276 milli sec,
        p90: 11.587 milli sec,
        p99: 11.587 milli sec,
        max: 11.587 milli sec,
        std_dev: 2681120.169;
      KV ingestion OPS: 370951  23740910 bytes/sec

    writebatch
      latency:
        mean: 0.008851280700000002,
        p50: 0.010000000000000002,
        p90: 0.022000000000000002,
        p99: 0.025,
        p100: 0.025;
      OPS: 112.97800102532054  13670338.124063784 bytes/sec

    getrandom
      latency:
        min: 21.148 micro sec,
        mean: 50.558 micro sec,
        p50: 41.604 micro sec,
        p90: 75.541 micro sec,
        p99: 134.044 micro sec,
        max: 251.834 micro sec,
        std_dev: 22059.745;
      QPS: 76426  7642615 bytes/sec

    getseq
      latency:
        min: 34.282 micro sec,
        mean: 44.888 micro sec,
        p50: 38.055 micro sec,
        p90: 62.057 micro sec,
        p99: 164.169 micro sec,
        max: 187.650 micro sec,
        std_dev: 20684.086;
      QPS: 86129  8612915 bytes/sec

    prefixscanrandom
      latency:
        min: 76.955 micro sec,
        mean: 119.778 micro sec,
        p50: 105.011 micro sec,
        p90: 172.588 micro sec,
        p99: 302.975 micro sec,
        max: 367.421 micro sec,
        std_dev: 42431.633;
      QPS: 32714  785148 bytes/sec

    deleterandom
      latency:
        min: 4.314 milli sec,
        mean: 4.314 milli sec,
        p50: 4.314 milli sec,
        p90: 4.314 milli sec,
        p99: 4.314 milli sec,
        max: 4.314 milli sec,
        std_dev: 0.000;
      KV ingestion OPS: 221178  13376905 bytes/sec

    deleterandom
      latency:
        mean: 0.00423134,
        p50: 0.005000000000000001,
        p90: 0.005000000000000001,
        p99: 0.005000000000000001,
        p100: 0.005000000000000001;
      OPS: 236.33175306167786  4690003.639508997 bytes/sec
STATISTICS:
state_store_get_latency P50 : 0.00003123048094940662 P95 : 0.00009410029498525074 P99 : 0.0002 P100 : 0.00025 COUNT : 2000 SUM : 0.09030594500000004
state_store_get_key_size P50 : 35 P95 : 48.5 P99 : 49.7 P100 : 50 COUNT : 2000 SUM : 42000
state_store_get_value_size P50 : 75 P95 : 97.5 P99 : 99.5 P100 : 100 COUNT : 2000 SUM : 200000
state_store_get_counts COUNT : 2000
state_store_get_snapshot_latency P50 : 0 P95 : 0 P99 : 0 P100 : 0 COUNT : 0 SUM : 0
state_store_range_scan_counts COUNT : 1000
state_store_reverse_range_scan_counts COUNT : 0
state_store_range_scan_size P50 : 1750 P95 : 2425 P99 : 2485 P100 : 2500 COUNT : 1000 SUM : 1210000
state_store_range_scan_latency P50 : 0.00025 P95 : 0.00047500000000000005 P99 : 0.000495 P100 : 0.0005 COUNT : 1000 SUM : 0.11525949000000002
state_store_batched_write_counts COUNT : 11
state_store_batched_write_tuple_counts COUNT : 10945
state_store_batched_write_latency P50 : 0.010000000000000002 P95 : 0.025 P99 : 0.025 P100 : 0.025 COUNT : 11 SUM : 0.09274414700000001
state_store_batched_write_size P50 : 150000 P95 : 200000 P99 : 200000 P100 : 200000 COUNT : 11 SUM : 1229845
state_store_batch_write_build_table_latency P50 : 0 P95 : 0 P99 : 0 P100 : 0 COUNT : 0 SUM : 0
state_store_batch_write_add_l0_ssts_latency P50 : 0 P95 : 0 P99 : 0 P100 : 0 COUNT : 0 SUM : 0
state_store_iter_counts COUNT : 0
state_store_iter_next_counts COUNT : 0
state_store_iter_seek_latency P50 : 0 P95 : 0 P99 : 0 P100 : 0 COUNT : 0 SUM : 0
state_store_iter_next_latency P50 : 0 P95 : 0 P99 : 0 P100 : 0 COUNT : 0 SUM : 0
state_store_iter_next_size P50 : 0 P95 : 0 P99 : 0 P100 : 0 COUNT : 0 SUM : 0
state_store_pin_version_counts COUNT : 0
state_store_unpin_version_counts COUNT : 0
state_store_pin_snapshot_counts COUNT : 0
state_store_unpin_snapshot_counts COUNT : 0
state_store_add_tables_counts COUNT : 0
state_store_get_new_table_id_counts COUNT : 0
state_store_get_compaction_task_counts COUNT : 0
state_store_report_compaction_task_counts COUNT : 0
state_store_pin_version_latency P50 : 0 P95 : 0 P99 : 0 P100 : 0 COUNT : 0 SUM : 0
state_store_unpin_version_latency P50 : 0 P95 : 0 P99 : 0 P100 : 0 COUNT : 0 SUM : 0
state_store_pin_snapshot_latency P50 : 0 P95 : 0 P99 : 0 P100 : 0 COUNT : 0 SUM : 0
state_store_unpin_snapshot_latency P50 : 0 P95 : 0 P99 : 0 P100 : 0 COUNT : 0 SUM : 0
state_store_add_tables_latency P50 : 0 P95 : 0 P99 : 0 P100 : 0 COUNT : 0 SUM : 0
state_store_get_new_table_id_latency P50 : 0 P95 : 0 P99 : 0 P100 : 0 COUNT : 0 SUM : 0
state_store_get_compaction_task_latency P50 : 0 P95 : 0 P99 : 0 P100 : 0 COUNT : 0 SUM : 0
state_store_report_compaction_task_latency P50 : 0 P95 : 0 P99 : 0 P100 : 0 COUNT : 0 SUM : 0
state_store_addtable_upload_sst_counts COUNT : 0
state_store_compaction_upload_sst_counts COUNT : 0

zehaowei · 2022-03-08T09:03:51Z

LGTM

twocode · 2022-03-09T04:49:31Z

rust/bench/ss_bench/main.rs

    statistics: bool,
+
+    #[clap(long)]
+    calibrate_metric: bool,


calibrate_histogram

Would fix this in the following PR

Sunt-ing added 26 commits March 1, 2022 17:30

basic impl

9612ee8

basic impl

3b0f329

basic impl

7da67a4

clippy

ae5d8bf

clippy

b011300

fmt

b1ab5c9

fmt

6234e30

fmt

2ab0d22

a little refactor

9839c16

a little refactor

6320eab

a little refactor

dfde376

a little refactor

d88da22

Merge branch 'main' into sunt_move_to_metric_system2

29d936f

Merge branch 'main' into sunt_move_to_metric_system2

03eecb5

impl myHistogram

d70c6fc

rm duplicated code

f44e6a8

Merge branch 'main' into sunt_move_to_metric_system2

0f106cc

fix bug: wrong init upper_bounds

e9ab89d

clippy

de08432

clean code

db44349

revert

2b72e39

refactor

646660b

refactor

c6dd7b5

refactor

025c417

refactor

4c01041

Merge branch 'main' into sunt_move_to_metric_system2

72c554e

Sunt-ing requested a review from twocode March 8, 2022 04:07

github-actions bot added the type/refactor Type: Refactoring. label Mar 8, 2022

twocode requested a review from zehaowei March 8, 2022 06:13

Sunt-ing added 2 commits March 8, 2022 16:54

adjust bucket

b1ea3f5

adjust bucket

bc17e0d

Sunt-ing changed the title ~~refactor(ss-bench): move to metrics system to display performance~~ feat(ss-bench): support calibrating metric parameter Mar 8, 2022

github-actions bot added type/feature Type: New feature. and removed type/refactor Type: Refactoring. labels Mar 8, 2022

Merge branch 'main' into sunt_move_to_metric_system2

6ea3025

rm dead code

4e269a8

Sunt-ing marked this pull request as ready for review March 8, 2022 09:06

twocode reviewed Mar 9, 2022

View reviewed changes

twocode approved these changes Mar 9, 2022

View reviewed changes

Sunt-ing merged commit 22cceb1 into main Mar 9, 2022

Sunt-ing deleted the sunt_move_to_metric_system2 branch March 9, 2022 04:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(ss-bench): support calibrating metric parameter #740

feat(ss-bench): support calibrating metric parameter #740

Uh oh!

Sunt-ing commented Mar 8, 2022 •

edited

Loading

Uh oh!

codecov bot commented Mar 8, 2022 •

edited

Loading

Uh oh!

Sunt-ing commented Mar 8, 2022 •

edited

Loading

Uh oh!

zehaowei commented Mar 8, 2022

Uh oh!

twocode Mar 9, 2022

Uh oh!

Sunt-ing Mar 9, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

feat(ss-bench): support calibrating metric parameter #740

feat(ss-bench): support calibrating metric parameter #740

Uh oh!

Conversation

Sunt-ing commented Mar 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What's changed and what's your intention?

Checklist

Refer to a related PR or issue link (optional)

Uh oh!

codecov bot commented Mar 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Sunt-ing commented Mar 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zehaowei commented Mar 8, 2022

Uh oh!

twocode Mar 9, 2022

Choose a reason for hiding this comment

Uh oh!

Sunt-ing Mar 9, 2022

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Sunt-ing commented Mar 8, 2022 •

edited

Loading

codecov bot commented Mar 8, 2022 •

edited

Loading

Sunt-ing commented Mar 8, 2022 •

edited

Loading