feat(sink): support create sink as query #6648

yuhao-su · 2022-11-29T08:13:02Z

I hereby agree to the terms of the Singularity Data, Inc. Contributor License Agreement.

What's changed and what's your intention?

support creating sink as a query
fix some wrongly assigned data in meta and missed handling for sink.

Checklist

I have written necessary rustdoc comments
I have added necessary unit tests and integration tests
All checks passed in ./risedev check (or alias, ./risedev c)

Documentation

If your pull request contains user-facing changes, please specify the types of the changes, and create a release note. Otherwise, please feel free to remove this section.

Types of user-facing changes

SQL commands, functions, and operators

Release note

support CREATE SINK sink_name AS select xxx from TABLE / MATERIALIZED VIEW / SOURCE WITH(connector = 'xxx')

Refer to a related PR or issue link (optional)

…hao/create_sink_as

src/frontend/src/optimizer/plan_node/stream_sink.rs

src/frontend/src/user/user_privilege.rs

src/meta/src/manager/catalog/mod.rs

HuaHuaY · 2022-11-29T09:09:33Z

src/meta/src/manager/catalog/mod.rs

        let user_core = &mut core.user;
        database_core.ensure_database_id(sink.database_id)?;
        database_core.ensure_schema_id(sink.schema_id)?;
-        database_core.ensure_table_id(sink.associated_table_id)?;


I suggest to check whether dependent_relations exists here.

None of the other streaming jobs did this. Maybe do it in a separate pr.

All of other streaming jobs do this. You can take a look at start_create_table_procedure. Some functions doesn't check dependent_relations, because we can find all the dependents without dependent_relations variable.

Oh I see. Acctually create_index did't do this. Should I also add the check for it?

No. Index only depends on its primary table, so we just check whether primary_table_id exists.

So the dependent_relations should contains 1 and only 1 item that is the primary_table_id ?

Yeah, index's dependent_relations only contains primary_table_id

Then let's add an assert here!

src/meta/src/manager/catalog/mod.rs

HuaHuaY · 2022-11-29T09:54:40Z

src/frontend/src/handler/create_sink.rs

+    // If colume names not specified, use the name in materialized view.
+    let col_names = get_column_names(&bound, session, stmt.columns)?.or(sink_col_names);

-    let sink_name = Binder::resolve_sink_name(stmt.sink_name)?;


Help to remove resolve_sink_name. Thanks

maybe no need to do that. still not sure if it will be used later

This resolve_sink_name was used to ensure that the sink and the corresponding table are in the same schema. I think after your PR, we don't need this constraint anymore. I think we support create sink xxx as select * from a.yyy join b.zzz on ... now and we have no idea whether xxx's schema is a or b

codecov · 2022-11-29T10:36:47Z

Codecov Report

Merging #6648 (1b62609) into main (f2801d3) will decrease coverage by 0.64%.
The diff coverage is 61.52%.

@@            Coverage Diff             @@
##             main    #6648      +/-   ##
==========================================
- Coverage   73.95%   73.30%   -0.65%     
==========================================
  Files         980     1012      +32     
  Lines      159120   162132    +3012     
==========================================
+ Hits       117671   118845    +1174     
- Misses      41449    43287    +1838

Flag	Coverage Δ
rust	`73.30% <61.52%> (-0.65%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/batch/src/error.rs	`0.00% <ø> (ø)`
src/batch/src/executor/group_top_n.rs	`68.42% <ø> (-6.44%)`	⬇️
src/batch/src/executor/insert.rs	`82.28% <ø> (ø)`
...batch/src/executor/join/distributed_lookup_join.rs	`0.00% <0.00%> (ø)`
src/batch/src/executor/join/nested_loop_join.rs	`91.73% <ø> (ø)`
src/batch/src/executor/join/sort_merge_join.rs	`79.04% <ø> (ø)`
src/batch/src/executor/mod.rs	`74.60% <0.00%> (+2.29%)`	⬆️
src/batch/src/executor/order_by.rs	`95.32% <ø> (ø)`
src/batch/src/executor/project_set.rs	`76.15% <ø> (ø)`
src/batch/src/executor/source.rs	`0.00% <0.00%> (ø)`
... and 438 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

src/meta/src/rpc/service/ddl_service.rs

src/meta/src/stream/stream_graph.rs

src/frontend/src/user/user_privilege.rs

proto/user.proto

tabVersion · 2022-11-30T05:59:10Z

e2e_test/sink/create_sink_as.slt

+CREATE SINK s4 AS select mv4.v1 as v1, mv4.v2 as v2 from mv4 WITH (
+    connector = 'mysql',
+    endpoint = '127.0.0.1:3306',
+    user = 'root',
+    database = 'test',
+    table = 't4'
+);


we are deprecating the built-in mysql connector soon. please use jdbc connector here.

blocked by #6631

How and when should I swith to jdbc?

Hi @yuhao-su, I think I can rewrite it in JDBC connector for you in #6631. In that PR I'm going to fix existing historic problems in sink e2e tests as well - it seems that sink e2e tests weren't enabled in CI at all.

e2e_test/sink/create_sink_as.slt

src/frontend/src/handler/create_mv.rs

proto/stream_plan.proto

src/meta/src/stream/stream_graph.rs

BugenZhao

Generally LGTM! Good work!

st1page

The Frontend part LGTM. But this PR modifies so many components, PTAL again 😿 c.c. @BugenZhao @yezizp2012 @HuaHuaY

HuaHuaY

LGTM!

yezizp2012

The meta part LGTM, great job!

yuhao-su added 5 commits November 28, 2022 14:55

create sink as

8be2f93

Merge branch 'main' of github.com:singularity-data/risingwave into yu…

1e27b42

…hao/create_sink_as

refactor

a7964c0

resolve conflict

53dfc2e

fmt

e878d00

yuhao-su requested review from BugenZhao, HuaHuaY, TennyZhuang, fuyufjh, nanderstabel, st1page and xxchan November 29, 2022 08:13

github-actions bot added the type/feature Type: New feature. label Nov 29, 2022

proto

b5e7c41

st1page reviewed Nov 29, 2022

View reviewed changes

src/frontend/src/optimizer/plan_node/stream_sink.rs Outdated Show resolved Hide resolved

src/frontend/src/optimizer/plan_node/stream_sink.rs Outdated Show resolved Hide resolved

src/frontend/src/optimizer/plan_node/stream_sink.rs Outdated Show resolved Hide resolved

proto fmt

d85311a

HuaHuaY reviewed Nov 29, 2022

View reviewed changes

resolve comments

e731693

yezizp2012 reviewed Nov 29, 2022

View reviewed changes

src/meta/src/rpc/service/ddl_service.rs Outdated Show resolved Hide resolved

src/meta/src/stream/stream_graph.rs Outdated Show resolved Hide resolved

src/frontend/src/user/user_privilege.rs Show resolved Hide resolved

proto/user.proto Show resolved Hide resolved

fix

33681ea

yuhao-su requested review from HuaHuaY, st1page and yezizp2012 November 30, 2022 05:28

yuhao-su added the user-facing-changes Contains changes that are visible to users label Nov 30, 2022

tabVersion reviewed Nov 30, 2022

View reviewed changes

StrikeW self-requested a review November 30, 2022 07:24

BugenZhao reviewed Nov 30, 2022

View reviewed changes

src/frontend/src/handler/create_mv.rs Outdated Show resolved Hide resolved

proto/stream_plan.proto Outdated Show resolved Hide resolved

src/meta/src/stream/stream_graph.rs Outdated Show resolved Hide resolved

rename

3f092ab

yuhao-su added 2 commits December 1, 2022 14:36

resolve conflict

b8cd8b6

remove table in sinknode

7ab95ca

BugenZhao reviewed Dec 1, 2022

View reviewed changes

dashboard

20bb41a

yuhao-su added the mergify/can-merge label Dec 1, 2022

st1page reviewed Dec 1, 2022

View reviewed changes

yuhao-su added 2 commits December 1, 2022 16:29

fix

8bcc519

another fix

2b36c81

HuaHuaY reviewed Dec 1, 2022

View reviewed changes

yezizp2012 reviewed Dec 1, 2022

View reviewed changes

st1page approved these changes Dec 1, 2022

View reviewed changes

Merge branch 'main' into yuhao/create_sink_as

1b62609

mergify bot merged commit fbe4dc6 into main Dec 1, 2022

mergify bot deleted the yuhao/create_sink_as branch December 1, 2022 09:17

KivenChen mentioned this pull request Dec 2, 2022

feat(ci): add e2e tests for connector sink and integrate sink e2e tests to ci #6631

Merged

3 tasks

hengm3467 mentioned this pull request Dec 20, 2022

Document create sink from a select query (create sink as) risingwavelabs/risingwave-docs-legacy#453

Closed

xiangjinwu mentioned this pull request Feb 26, 2024

fix(frontend): index if not exists check shall use table's schema #15249

Merged

9 tasks

feat(sink): support create sink as query #6648

feat(sink): support create sink as query #6648

Uh oh!

Conversation

yuhao-su commented Nov 29, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What's changed and what's your intention?

Checklist

Documentation

Types of user-facing changes

Release note

Refer to a related PR or issue link (optional)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

HuaHuaY Nov 29, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HuaHuaY Nov 29, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yuhao-su Nov 29, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HuaHuaY Nov 29, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Nov 29, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BugenZhao left a comment

Choose a reason for hiding this comment

Uh oh!

st1page left a comment

Choose a reason for hiding this comment

Uh oh!

HuaHuaY left a comment

Choose a reason for hiding this comment

Uh oh!

yezizp2012 left a comment

Choose a reason for hiding this comment

yuhao-su commented Nov 29, 2022 •

edited

Loading

HuaHuaY Nov 29, 2022 •

edited

Loading

HuaHuaY Nov 29, 2022 •

edited

Loading

yuhao-su Nov 29, 2022 •

edited

Loading

HuaHuaY Nov 29, 2022 •

edited

Loading

codecov bot commented Nov 29, 2022 •

edited

Loading