feat(mview): support barrier from meta #88
Conversation
|
feat(mview) :eat_watermelon |
rust/meta/src/barrier/mod.rs
Outdated
| // TODO(MrCroxx): refine this | ||
| let mut actor_ids_to_collect = info.actor_ids_to_collect(node_id).collect_vec(); | ||
| match &mutation { | ||
| Mutation::Add(AddMutation { actors }) => { | ||
| if !actors.is_empty() { | ||
| actor_ids_to_collect.extend(new_actors_to_collect.drain(..)); | ||
| } | ||
| } | ||
| _ => {} | ||
| } |
There was a problem hiding this comment.
@BugenZhao Collecting from newly created actors may not be a good idea. Maybe still collect from existing actor and let chain executor swallow the first barrier (conf change barrier) and get its epoch is better?
There was a problem hiding this comment.
I've no idea which approach is better. Considering there'll still be some changes if we are to optimize large snapshot ingesting, let's make it run first!
There was a problem hiding this comment.
IMO, if we collect barriers from newly created actors, we need to collect them in any cases, not only for mv on mv, which haven't been implemented now.
There was a problem hiding this comment.
I have discussed with @fuyufjh before, we prefer not to collect barriers from newly created actors(not caught up actors). Both swallow in chain or ignore the old epoch when collecting after actors ready is both okay. Swallow it would be more simple in any cases.
There was a problem hiding this comment.
Not collect from newly created actors will also not block other normal barriers, like checkpoint(flush).
There was a problem hiding this comment.
Not collect from newly created actors will also not block other normal barriers, like checkpoint(flush).
What about setting actor status "ready"? Will the operation needs a barrier? If needed, should that barrier be collected from the new ready actors? That case is more like MV on MV, which barrier only updates the outputs of the dispatchers.
There was a problem hiding this comment.
For CreateMaterializedView cmd, the barrier should be collected from the new created actors. For MV on MV or MV on table which already contains large amount of data, the creating operation should be blocked, which means Chain actor should hold the creating barrier, when almost catch up, it should pass the barrier and let it be collected by barrier manager. Until now, the creating cmd finished and the status will be set to ready.
Codecov Report
@@ Coverage Diff @@
## main #88 +/- ##
============================================
+ Coverage 74.40% 74.50% +0.09%
Complexity 2645 2645
============================================
Files 829 829
Lines 47563 47609 +46
Branches 1560 1560
============================================
+ Hits 35390 35469 +79
+ Misses 11375 11342 -33
Partials 798 798
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
What's changed and what's your intention?
Adapt create materialized view to the meta barrier manager. All barrier and meta store operations will be moved to barrier manager.
Currently MV on MV does not support parallel mview upstream (#58) and some other features in this PR. I will implement them in another some PRs. So MV on MV e2e still cannot run. I've checked the e2e test with parallel degree equals 1 and works.
Checklist
Refer to a related PR or issue link (optional)
#87