Skip to content

Conversation

@yezizp2012
Copy link
Member

@yezizp2012 yezizp2012 commented Feb 9, 2022

What's changed and what's your intention?

As title, for shared executor, operator_id generated in frontend is not globally unique(only unique since server start within itself), we should use fragment_id+operator_id as its keyspace.

Checklist

  • I have written necessary docs and comments
  • I have added necessary unit tests and integration tests

Refer to a related PR or issue link (optional)

@github-actions github-actions bot added the type/fix Type: Bug fix. Only for pull requests. label Feb 9, 2022
@BugenZhao BugenZhao changed the title fix: fix shared executor keyspace fix(streaming): shared executor keyspace Feb 9, 2022
@codecov
Copy link

codecov bot commented Feb 9, 2022

Codecov Report

Merging #156 (b81dddf) into main (cabaee0) will increase coverage by 0.00%.
The diff coverage is 69.23%.

Impacted file tree graph

@@            Coverage Diff            @@
##               main     #156   +/-   ##
=========================================
  Coverage     74.65%   74.65%           
  Complexity     2654     2654           
=========================================
  Files           834      834           
  Lines         47883    47888    +5     
  Branches       1562     1562           
=========================================
+ Hits          35747    35752    +5     
  Misses        11337    11337           
  Partials        799      799           
Flag Coverage Δ
java 62.08% <ø> (ø)
rust 80.07% <69.23%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
rust/storage/src/keyspace.rs 85.45% <0.00%> (ø)
rust/stream/src/task/test_mv.rs 100.00% <ø> (ø)
rust/stream/src/task/tests.rs 100.00% <ø> (ø)
rust/stream/src/task/stream_manager.rs 52.47% <88.88%> (+0.45%) ⬆️
rust/meta/src/stream/graph/stream_graph.rs 90.90% <100.00%> (+0.13%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cabaee0...b81dddf. Read the comment docs.

Copy link
Member

@BugenZhao BugenZhao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about directly assembling a real operator_id here, so that we can only pass a single parameters to initialization of Keyspace?

        // We assume that the operator_id of different instances from the same RelNode will be the
        // same.
        let executor_id = ((actor_id as u64) << 32)    + node.get_operator_id();
+       let operator_id = ((fragment_id as u64) << 32) + node.get_operator_id();
-       let operator_id = node.get_operator_id().try_into().unwrap();

@yezizp2012
Copy link
Member Author

How about directly assembling a real operator_id here, so that we can only pass a single parameters to initialization of Keyspace?

        // We assume that the operator_id of different instances from the same RelNode will be the
        // same.
        let executor_id = ((actor_id as u64) << 32)    + node.get_operator_id();
+       let operator_id = ((fragment_id as u64) << 32) + node.get_operator_id();
-       let operator_id = node.get_operator_id().try_into().unwrap();

Using operator_id as parameter in shared_executor_root is a little bit confusing, and i don't think of a better naming for it.

@BugenZhao
Copy link
Member

How about directly assembling a real operator_id here, so that we can only pass a single parameters to initialization of Keyspace?

        // We assume that the operator_id of different instances from the same RelNode will be the
        // same.
        let executor_id = ((actor_id as u64) << 32)    + node.get_operator_id();
+       let operator_id = ((fragment_id as u64) << 32) + node.get_operator_id();
-       let operator_id = node.get_operator_id().try_into().unwrap();

Using operator_id as parameter in shared_executor_root is a little bit confusing, and i don't think of a better naming for it.

Actually node.get_operator_id() generated by the frontend is not what we want for operator_id, it's more like an "offset" which is only meaningful when we combine it with actor_id or fragment_id. Keyspace may not care about what exactly its parameters are for, it just wants to accept a unique ID corresponding to its RelNode.

@yezizp2012
Copy link
Member Author

How about directly assembling a real operator_id here, so that we can only pass a single parameters to initialization of Keyspace?

        // We assume that the operator_id of different instances from the same RelNode will be the
        // same.
        let executor_id = ((actor_id as u64) << 32)    + node.get_operator_id();
+       let operator_id = ((fragment_id as u64) << 32) + node.get_operator_id();
-       let operator_id = node.get_operator_id().try_into().unwrap();

Using operator_id as parameter in shared_executor_root is a little bit confusing, and i don't think of a better naming for it.

Actually node.get_operator_id() generated by the frontend is not what we want for operator_id, it's more like an "offset" which is only meaningful when we combine it with actor_id or fragment_id. Keyspace may not care about what exactly its parameters are for, it just wants to accept a unique ID corresponding to its RelNode.

Well, if we treat operator_id as an offset, it makes sense here. I will change it.

@yezizp2012 yezizp2012 enabled auto-merge (squash) February 9, 2022 06:08
@yezizp2012 yezizp2012 merged commit f395dee into main Feb 9, 2022
@yezizp2012 yezizp2012 deleted the fix/shared-executor-keyspace branch February 9, 2022 06:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type/fix Type: Bug fix. Only for pull requests.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants