Skip to content

Conversation

@xx01cyx
Copy link
Contributor

@xx01cyx xx01cyx commented Feb 14, 2023

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

The type of the sink (append-only/upsert) can be deduced in the frontend, and we can simply pass this type to computing. The deduction requires 3 pieces of information:

  1. The append-only attribute derived by the optimizer
  2. The append-only attribute specified by the user: CREATE SINK s FROM t WITH (...., format = 'append_only')
  3. The force-append-only attribute specified by the user: CREATE SINK s FROM t WITH (...., force_append_only = 'true')

There are 2 * 2 * 2 = 8 combinations of these 3 attributes. The figure below elaborates on the combinations and their results.

image

Checklist

  • I have written necessary rustdoc comments
  • I have added necessary unit tests and integration tests
  • All checks passed in ./risedev check (or alias, ./risedev c)

Related issue

Closes #7414

Documentation

Types of user-facing changes

Please keep the types that apply to your changes, and remove the others.

  • Connector (sources & sinks)
  • SQL commands, functions, and operators

Release note

The user will get an error if the sink has format = append_only in with options while it actually cannot be append-only.

=> CREATE TABLE t (v int);   # t is NOT append-only
CREATE_TABLE

=> CREATE SINK s FROM t WITH (
    connector = 'kafka',
    format = 'append_only',
    ......
);
ERROR:  QueryError: Sink error: The sink cannot be append-only. Please add "force_append_only='true'" in WITH options to force the sink to be append-only. Notice that this will cause the sink executor to drop any UPDATE or DELETE message.

The user can fix this by enforcing the sink to be append-only:

=> CREATE TABLE t (v int);   # t is NOT append-only
CREATE_TABLE

=> CREATE SINK s FROM t WITH (
    connector = 'kafka',
    format = 'append_only',
    force_append_only = 'true',
    ......
);
CREATE_SINK

@xx01cyx xx01cyx marked this pull request as draft February 14, 2023 10:02
@github-actions github-actions bot added the type/feature Type: New feature. label Feb 14, 2023
@xx01cyx xx01cyx marked this pull request as ready for review February 14, 2023 13:36
@xx01cyx xx01cyx force-pushed the cyx/sink-append-only-derivation branch from edb0ca5 to 4e92427 Compare February 14, 2023 15:21
@codecov
Copy link

codecov bot commented Feb 15, 2023

Codecov Report

Merging #7922 (9c0bedc) into main (7f13a0a) will increase coverage by 0.00%.
The diff coverage is 65.78%.

@@           Coverage Diff           @@
##             main    #7922   +/-   ##
=======================================
  Coverage   71.61%   71.61%           
=======================================
  Files        1116     1116           
  Lines      179579   179604   +25     
=======================================
+ Hits       128607   128631   +24     
- Misses      50972    50973    +1     
Flag Coverage Δ
rust 71.61% <65.78%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/connector/src/sink/mod.rs 0.00% <ø> (ø)
src/stream/src/executor/sink.rs 0.00% <0.00%> (ø)
...tend/src/optimizer/plan_node/stream_materialize.rs 90.47% <55.55%> (-3.95%) ⬇️
src/connector/src/sink/kafka.rs 37.08% <100.00%> (ø)
src/frontend/src/optimizer/mod.rs 95.26% <100.00%> (ø)
src/frontend/src/utils/with_options.rs 86.95% <100.00%> (+0.59%) ⬆️
...frontend/src/scheduler/hummock_snapshot_manager.rs 60.00% <0.00%> (-0.48%) ⬇️
src/object_store/src/object/mem.rs 86.87% <0.00%> (-0.39%) ⬇️
src/object_store/src/object/mod.rs 51.19% <0.00%> (-0.22%) ⬇️
src/storage/src/hummock/sstable_store.rs 65.16% <0.00%> (-0.16%) ⬇️
... and 7 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@xx01cyx xx01cyx added the user-facing-changes Contains changes that are visible to users label Feb 15, 2023
Copy link
Contributor

@tabVersion tabVersion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, good work!

just curious, how we deduce whether an mv is append-only

@xx01cyx
Copy link
Contributor Author

xx01cyx commented Feb 15, 2023

just curious, how we deduce whether an mv is append-only

Each streaming operator in the mv plan will derive its append-only attribute according to its input operator's append-only attribute and its own streaming behavior. And an mv's append-only attribute is exactly the materialize operator's append-only attribute.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type/feature Type: New feature. user-facing-changes Contains changes that are visible to users

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tracking: sink refactor for separating append-only and non-append-only

4 participants