Skip to content

feat(meta): alter streaming schedule strategy #940

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 23 commits into from
Mar 16, 2022
Merged

feat(meta): alter streaming schedule strategy #940

merged 23 commits into from
Mar 16, 2022

Conversation

@xx01cyx
Copy link
Contributor

@xx01cyx xx01cyx commented Mar 15, 2022

What's changed and what's your intention?

Please explain IN DETAIL what the changes are in this PR and why they are needed:

  • Summarize your change
    • Alter streaming schedule strategy by introducing parallel units.
    • Refactor streaming scheduler. Now the scheduler performs scheduling on fragment instead of actors.

Checklist

  • I have written necessary docs and comments
  • I have added necessary unit tests and integration tests

Refer to a related PR or issue link

#670
#858

xx01cyx added 16 commits March 8, 2022 13:53
@xx01cyx
feat(meta): distinguish and assign responsibilities for different par…
80120c6 
…allel units (simple dispatch and hash dispatch)
@xx01cyx
Merge branch 'main' into feat/schedule
0368070
@xx01cyx
chore: create parallel units only when the newly added node is CN
d41eb43
@xx01cyx
chore: fix test for HashDispatcherManager
139bb66
@xx01cyx
fix: alter allocation strategy for single and hash parallel units
229cfd0
@xx01cyx
merge main into feat/schedule
24159cd
@xx01cyx
refactor: remove dashmap and add core in StoredClusterManager
71b868d
@xx01cyx
merge main into feat/schedule
4bb5e46
@xx01cyx
chore: fix typo
476d828
@xx01cyx
merge & refactor: merge main into feat/schedule and refactor StoredCl…
ad1d37f 
…usterManager
@xx01cyx
Merge branch 'main' into feat/schedule
e6b86a5
@xx01cyx
feat(meta): alter schedule strategy for fragments
95c6cfd
@xx01cyx
Merge branch 'main' into feat/schedule
0561bee
@xx01cyx
Merge branch 'main' into feat/schedule
5442d2d
@xx01cyx
chore: fmt rust code
7eb4d81
@xx01cyx
chore: fix test and add doc
1327ab7
@xx01cyx xx01cyx requested review from fuyufjh and yezizp2012 and removed request for yezizp2012 March 15, 2022 10:24
@codecov
Copy link

codecov bot commented Mar 15, 2022 •
edited
Loading

Codecov Report

Merging #940 (bc37156) into main (58da156) will decrease coverage by 0.12%.
The diff coverage is 81.20%.

@@             Coverage Diff              @@
##               main     #940      +/-   ##
============================================
- Coverage     71.80%   71.68%   -0.13%     
  Complexity     2766     2766              
============================================
  Files           971      972       +1     
  Lines         56773    56927     +154     
  Branches       1787     1787              
============================================
+ Hits          40767    40809      +42     
- Misses        15116    15228     +112     
  Partials        890      890
Flag Coverage Δ
java 61.23% <ø> (ø)
rust 75.69% <81.20%> (-0.19%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
rust/meta/src/rpc/service/stream_service.rs 100.00% <ø> (ø)
rust/meta/src/cluster/mod.rs 70.50% <34.78%> (-3.35%) ⬇️
rust/meta/src/stream/stream_manager.rs 74.00% <76.47%> (+0.05%) ⬆️
rust/meta/src/stream/scheduler.rs 93.45% <92.68%> (+2.79%) ⬆️
rust/frontend/src/scheduler/plan_fragmenter.rs 97.31% <100.00%> (-0.03%) ⬇️
rust/meta/src/manager/hash_dispatch.rs 90.24% <100.00%> (+0.12%) ⬆️
rust/meta/src/model/stream.rs 72.05% <100.00%> (+0.84%) ⬆️
rust/meta/src/stream/fragmenter.rs 80.43% <100.00%> (ø)
rust/meta/src/stream/test_fragmenter.rs 98.96% <100.00%> (+0.28%) ⬆️
...ream/src/executor/managed_state/aggregation/mod.rs 76.59% <0.00%> (-3.41%) ⬇️
... and 14 more

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

@xx01cyx xx01cyx changed the title Feat/schedule feat(meta): alter streaming schedule strategy Mar 15, 2022
@github-actions github-actions bot added the type/feature Type: New feature. label Mar 15, 2022
fuyufjh
fuyufjh approved these changes Mar 16, 2022
View reviewed changes
Copy link
Collaborator

@fuyufjh fuyufjh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for the rest

yezizp2012
yezizp2012 approved these changes Mar 16, 2022
View reviewed changes
Copy link
Member

@yezizp2012 yezizp2012 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally LGTM

@skyzh
Copy link
Contributor

skyzh commented Mar 16, 2022 •
edited
Loading

can we replace node_id with worker_node_id? There used to be another nodeId in proto representing the id of executor in each fragment, which might be confusing...

@xx01cyx xx01cyx merged commit 4255938 into main Mar 16, 2022
@xx01cyx xx01cyx deleted the feat/schedule branch March 16, 2022 07:30
@xx01cyx xx01cyx mentioned this pull request Mar 18, 2022
Merged
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fuyufjh fuyufjh fuyufjh approved these changes

@yezizp2012 yezizp2012 yezizp2012 approved these changes

Assignees

No one assigned

Labels

type/feature Type: New feature.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@xx01cyx @skyzh @fuyufjh @yezizp2012