Skip to content

feat(optimizer): add FilterProjectRule #945

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Mar 17, 2022
Merged

feat(optimizer): add FilterProjectRule #945

merged 6 commits into from
Mar 17, 2022

Conversation

@xxchan
Copy link
Collaborator

@xxchan xxchan commented Mar 15, 2022 •
edited
Loading

What's changed and what's your intention?

  • add FilterProjectRule
  • Fixed the assert error when calling LogicalProject::with_mapping: should use with_capacity to create a FixedBitSet with explicit capacity instead of collect.

Checklist

  • I have written necessary docs and comments
  • I have added necessary unit tests and integration tests

Refer to a related PR or issue link (optional)

part of #842

@github-actions github-actions bot added the type/feature Type: New feature. label Mar 15, 2022
@xxchan
Copy link
Collaborator Author

xxchan commented Mar 15, 2022

unable to trigger it before supporting subquery? e.g., select v from (select v from t) where v > 1

@xxchan xxchan marked this pull request as ready for review March 15, 2022 15:00
@codecov
Copy link

codecov bot commented Mar 15, 2022 •
edited
Loading

Codecov Report

Merging #945 (9fd680b) into main (6b044f6) will increase coverage by 0.40%.
The diff coverage is 33.33%.

❗ Current head 9fd680b differs from pull request most recent head 49a673c. Consider uploading reports for the commit 49a673c to get more accurate results

@@             Coverage Diff              @@
##               main     #945      +/-   ##
============================================
+ Coverage     71.43%   71.84%   +0.40%     
  Complexity     2766     2766              
============================================
  Files           978      972       -6     
  Lines         57824    56609    -1215     
  Branches       1790     1787       -3     
============================================
- Hits          41308    40668     -640     
+ Misses        15625    15051     -574     
+ Partials        891      890       -1
Flag Coverage Δ
java 61.23% <ø> (+0.19%) ⬆️
rust 75.93% <33.33%> (+0.56%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...rontend/src/optimizer/plan_node/logical_project.rs 97.51% <ø> (+0.50%) ⬆️
rust/frontend/src/optimizer/rule/filter_project.rs 0.00% <0.00%> (ø)
rust/frontend/src/utils/mod.rs 83.33% <83.33%> (ø)
rust/storage/src/hummock/state_store_tests.rs 0.00% <0.00%> (-100.00%) ⬇️
rust/frontend/src/optimizer/property/convention.rs 0.00% <0.00%> (-100.00%) ⬇️
...ust/frontend/src/optimizer/plan_node/batch_sort.rs 0.00% <0.00%> (-100.00%) ⬇️
...ontend/src/optimizer/plan_node/stream_hash_join.rs 0.00% <0.00%> (-75.00%) ⬇️
rust/batch/src/task/channel.rs 0.00% <0.00%> (-60.00%) ⬇️
...rontend/src/optimizer/plan_node/stream_exchange.rs 0.00% <0.00%> (-48.15%) ⬇️
rust/meta/src/hummock/vacuum.rs 39.13% <0.00%> (-47.45%) ⬇️
... and 196 more

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

@fuyufjh
Copy link
Collaborator

fuyufjh commented Mar 15, 2022

unable to trigger it before supporting subquery? e.g., select v from (select v from t) where v > 1

Please verify it with unit tests

@fuyufjh
Copy link
Collaborator

fuyufjh commented Mar 17, 2022

Can you please add some plan tests in predicate_pushdown.yaml? LSTM for the rest.

@xxchan xxchan marked this pull request as draft March 17, 2022 07:19
@xxchan xxchan marked this pull request as ready for review March 17, 2022 11:41
@xxchan
Copy link
Collaborator Author

xxchan commented Mar 17, 2022

Fixed the assert error when calling LogicalProject::with_mapping: should use with_capacity to create a FixedBitSet with explicit capacity instead of collect.

This was not triggered before, because the largest index was always selected...

fuyufjh
fuyufjh approved these changes Mar 17, 2022
View reviewed changes
rust/frontend/test_runner/tests/testdata/column_pruning.yaml
logical_plan: |
LogicalProject { exprs: [$1], expr_alias: [Some("v1")] }
LogicalScan { table: "t", columns: ["_row_id", "v1", "v2"] }
batch_plan: |
Copy link
Collaborator

@fuyufjh fuyufjh Mar 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to test only logical_plan here?

Copy link
Collaborator Author

@xxchan xxchan Mar 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had this idea too. We can split logical optimzation from gen_batch_query_plan (in a next PR).

But in tests, which one do you think is better:

  1. logical_plan is logically optimized plan.
  2. add a separate logical_plan_optimized

Copy link
Collaborator

@fuyufjh fuyufjh Mar 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prefer optimized_logical_plan

https://spark.apache.org/docs/latest/sql-ref-syntax-qry-explain.html

@xxchan xxchan mentioned this pull request Mar 17, 2022
Closed
@xxchan xxchan merged commit 4db91cf into main Mar 17, 2022
@xxchan xxchan deleted the filter_project branch March 17, 2022 12:15
@xxchan xxchan mentioned this pull request Mar 23, 2022
Closed
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fuyufjh fuyufjh fuyufjh approved these changes

@st1page st1page Awaiting requested review from st1page

@xiangjinwu xiangjinwu Awaiting requested review from xiangjinwu

Assignees

No one assigned

Labels

type/feature Type: New feature.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@xxchan @fuyufjh