feat(stream): support parallel source #232
Conversation
| // Currently, we assume the parallel degree is at least 4, and grows linearly with | ||
| // more worker nodes added. | ||
| max(self.worker_count * 2, PARALLEL_DEGREE_LOW_BOUND) | ||
| } else { |
There was a problem hiding this comment.
This will set source parallel degree same to others in the middle. Previously the source degree is set to worker count, @wyhyhyhyh added a logic to make them round robin at every worker node. You may need to fix the test test_fragmenter_case2 in test_fragmenter.rs.
There was a problem hiding this comment.
In my understanding, for table V2, source degree is not necessary equivalent with worker count. But anyway source has to be coherent with the MV number, and colocated with MV if necessary?
Codecov Report
@@ Coverage Diff @@
## main #232 +/- ##
============================================
- Coverage 74.08% 74.07% -0.01%
Complexity 2665 2665
============================================
Files 850 850
Lines 48083 48079 -4
Branches 1584 1584
============================================
- Hits 35622 35616 -6
- Misses 11651 11653 +2
Partials 810 810
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
What's changed and what's your intention?
Support parallel source, especially the table source V2. After this PR:
(Waiting for E2E to see whether Kafka source works...)
Checklist
Refer to a related PR or issue link (optional)
Closes #111