Skip to content

test(source): add ci e2e test for mysql-cdc #6481

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 13 commits into from
Nov 22, 2022
Merged

test(source): add ci e2e test for mysql-cdc #6481

merged 13 commits into from
Nov 22, 2022

Conversation

@StrikeW
Copy link
Contributor

@StrikeW StrikeW commented Nov 21, 2022 •
edited
Loading

I hereby agree to the terms of the Singularity Data, Inc. Contributor License Agreement.

What's changed and what's your intention?

Add e2e test for MySQL CDC (MySQL --> connector node --> RisingWave) to PR pipeline.
The test cases are from Flink's tutorial.

  | e2e_test/source/cdc/cdc.load.slt .. [OK] in 1977 ms
  | e2e_test/source/cdc/cdc.check.slt .. [OK] in 64 ms

Checklist

  • I have written necessary rustdoc comments
  • I have added necessary unit tests and integration tests
  • All checks passed in ./risedev check (or alias, ./risedev c)

Documentation

If your pull request contains user-facing changes, please specify the types of the changes, and create a release note. Otherwise, please feel free to remove this section.

Types of user-facing changes

Please keep the types that apply to your changes, and remove those that do not apply.

  • Installation and deployment
  • Connector (sources & sinks)
  • SQL commands, functions, and operators
  • RisingWave cluster configuration changes
  • Other (please specify in the release note below)

Release note

Please create a release note for your changes. In the release note, focus on the impact on users, and mention the environment or conditions where the impact may occur.

Refer to a related PR or issue link (optional)

#6345

@github-actions github-actions bot added the type/feature Type: New feature. label Nov 21, 2022
@StrikeW StrikeW changed the title feat(source): add ci e2e test for mysql-cdc test(source): add ci e2e test for mysql-cdc Nov 21, 2022
@github-actions github-actions bot added A-test Area: Testing framework & misc tests not belonging to any specific component. and removed type/feature Type: New feature. labels Nov 21, 2022
@StrikeW StrikeW force-pushed the siyuan/cdc-source branch 3 times, most recently from 2fc5ed0 to e96d01c Compare November 21, 2022 12:35
@StrikeW StrikeW force-pushed the siyuan/cdc-source branch 8 times, most recently from 106b86f to bb79677 Compare November 21, 2022 16:44
@codecov
Copy link

codecov bot commented Nov 22, 2022 •
edited
Loading

Codecov Report

Merging #6481 (8fc00ea) into main (213da6a) will decrease coverage by 0.06%.
The diff coverage is 54.10%.

@@            Coverage Diff             @@
##             main    #6481      +/-   ##
==========================================
- Coverage   74.01%   73.95%   -0.07%     
==========================================
  Files         983      986       +3     
  Lines      160712   161356     +644     
==========================================
+ Hits       118952   119329     +377     
- Misses      41760    42027     +267
Flag Coverage Δ
rust 73.95% <54.10%> (-0.07%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/batch/src/executor/mod.rs 72.30% <ø> (ø)
src/batch/src/executor/source.rs 0.00% <0.00%> (ø)
src/common/src/util/prost.rs 0.00% <0.00%> (ø)
src/compute/src/rpc/service/stream_service.rs 0.00% <0.00%> (ø)
src/frontend/src/observer/observer_manager.rs 0.00% <0.00%> (ø)
...c/frontend/src/optimizer/plan_node/batch_source.rs 0.00% <0.00%> (ø)
src/frontend/src/optimizer/plan_node/mod.rs 85.49% <ø> (ø)
src/frontend/src/optimizer/plan_node/stream.rs 13.24% <0.00%> (-0.07%) ⬇️
src/frontend/src/scheduler/distributed/stage.rs 19.92% <0.00%> (-1.09%) ⬇️
src/frontend/src/scheduler/local.rs 0.00% <0.00%> (ø)
... and 75 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@StrikeW StrikeW requested review from tabVersion and wenym1 November 22, 2022 02:07
@StrikeW StrikeW marked this pull request as ready for review November 22, 2022 02:07
huangjw806
huangjw806 approved these changes Nov 22, 2022
View reviewed changes
Copy link
Contributor

@huangjw806 huangjw806 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

fuyufjh
fuyufjh approved these changes Nov 22, 2022
View reviewed changes
Copy link
Collaborator

@fuyufjh fuyufjh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

BugenZhao
BugenZhao reviewed Nov 22, 2022
View reviewed changes
wenym1
wenym1 reviewed Nov 22, 2022
View reviewed changes
ci/scripts/e2e-source-test.sh
# start risingwave cluster
cargo make ci-start ci-1cn-1fe
# start cdc connector node
nohup java -cp ./connector-service.jar com.risingwave.sourcenode.service.SourceServiceMain > .risingwave/log/connector-source.log 2>&1 &
Copy link
Contributor

@wenym1 wenym1 Nov 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this future when we combine the sink and source service into a single service, we only need java -jar ./connector-service.jar.

Copy link
Contributor Author

@StrikeW StrikeW Nov 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correct

BugenZhao
BugenZhao reviewed Nov 22, 2022
View reviewed changes
liurenjie1024
liurenjie1024 approved these changes Nov 22, 2022
View reviewed changes
Copy link
Contributor

@liurenjie1024 liurenjie1024 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

BugenZhao
BugenZhao reviewed Nov 22, 2022
View reviewed changes
@mergify mergify bot merged commit ca95541 into main Nov 22, 2022
@mergify mergify bot deleted the siyuan/cdc-source branch November 22, 2022 08:35
@wenym1 wenym1 mentioned this pull request Nov 22, 2022
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@BugenZhao BugenZhao BugenZhao left review comments

@wenym1 wenym1 wenym1 left review comments

@fuyufjh fuyufjh fuyufjh approved these changes

@neverchanje neverchanje Awaiting requested review from neverchanje

@tabVersion tabVersion Awaiting requested review from tabVersion

+2 more reviewers

@liurenjie1024 liurenjie1024 liurenjie1024 approved these changes

@huangjw806 huangjw806 huangjw806 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

A-test Area: Testing framework & misc tests not belonging to any specific component.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@StrikeW @liurenjie1024 @fuyufjh @BugenZhao @huangjw806 @wenym1