feat: add barrier latency monitor #553
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
skyzh
reviewed
Feb 25, 2022
rust/meta/src/rpc/metrics.rs
Outdated
| } | ||
| lazy_static::lazy_static! { | ||
| pub static ref | ||
| DEFAULT_META_STATS: Arc<MetaMetrics> = Arc::new(MetaMetrics::new()); |
Contributor
There was a problem hiding this comment.
Don't use lazy_static in meta metrics. The metrics in meta service is designed to not affect the global scope. We should also later refactor compute node.
BugenZhao
reviewed
Feb 25, 2022
rust/meta/src/barrier/mod.rs
Outdated
|
|
||
| /// When `buffer` is not empty anymore, all subscribers of this watcher will be notified. | ||
| changed_tx: watch::Sender<()>, | ||
| metrics: Arc<MetaMetrics>, |
Member
There was a problem hiding this comment.
Should move to BarrierManager.
skyzh
approved these changes
Feb 25, 2022
| let buckets = BARRIER_BUCKETS; | ||
| let opts = histogram_opts!( | ||
| "meta_barrier_duration_seconds", | ||
| "barrier latency ", |
rust/meta/src/rpc/metrics.rs
Outdated
| let opts = histogram_opts!( | ||
| "meta_barrier_duration_seconds", | ||
| "barrier latency ", | ||
| buckets.iter().map(|x| *x * 0.1).collect_vec() |
Contributor
There was a problem hiding this comment.
As we define BARRIER_BUCKETS manually, do we still need to *0.1 here?
rust/meta/src/rpc/metrics.rs
Outdated
| "meta_grpc_duration_seconds", | ||
| "gRPC latency of meta services", | ||
| buckets.iter().map(|x| *x * 0.1).collect_vec() | ||
| buckets.iter().map(|x| *x * 1.0).collect_vec() |
|
|
||
| /// gRPC latency of meta services | ||
| pub grpc_latency: HistogramVec, | ||
| pub barrier_latency: Histogram, |
Contributor
There was a problem hiding this comment.
Add docstring for this field.
Contributor
|
Suggest attaching a grafana/tracing screenshot within a PR regarding monitoring/metrics. 😄 |
yezizp2012
pushed a commit
that referenced
this pull request
Feb 28, 2022
* add barrier monitor * fix * fix * x * x
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What's changed and what's your intention?
PLEASE DO NOT LEAVE THIS EMPTY !!!
Please explain IN DETAIL what the changes are in this PR and why they are needed:
Summarize your change (mandatory)
This PR adds barrier latency monitor, and updates the corresponding grafana dashboards.
How does this PR work? Need a brief introduction for the changed logic (optional)
Describe clearly one logical change and avoid lazy messages (optional)
Describe any limitations of the current code (optional)
Checklist
Refer to a related PR or issue link (optional)