Skip to content

feat(source): add s3 source and csv parser #6846

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Dec 21, 2022
Merged

feat(source): add s3 source and csv parser #6846

merged 2 commits into from
Dec 21, 2022

Conversation

@waruto210
Copy link
Contributor

@waruto210 waruto210 commented Dec 11, 2022 •
edited
Loading

I hereby agree to the terms of the Singularity Data, Inc. Contributor License Agreement.

What's changed and what's your intention?

PLEASE DO NOT LEAVE THIS EMPTY !!!

  • Add abstraction of file system data sources
  • Add S3 source and csv parser

Checklist

  • I have written necessary rustdoc comments
  • I have added necessary unit tests and integration tests
  • All checks passed in ./risedev check (or alias, ./risedev c)

Documentation

Types of user-facing changes

Please keep the types that apply to your changes, and remove those that do not apply.

  • Connector (sources & sinks)

Release note

create materialized source s(
    id int,
    name varchar,
    age int,
    primary key(id)
) with (
    connector = 's3',
    s3.region_name = 'ap-southeast-1',
    s3.bucket_name = 'mingchao-s3-source',
    s3.credentials.access = 'xxx',
    s3.credentials.secret = 'xxx'
) row format csv [WITHOUT HEADER] delimited by ',';

csv file

id,name,age
1,alice,22
2,mike,23

It's necessary to have a line break after each row, including the last line.

Refer to a related PR or issue link (optional)

#6511

@github-actions github-actions bot added the type/feature Type: New feature. label Dec 11, 2022
@waruto210 waruto210 force-pushed the waruto/s3-source branch 2 times, most recently from f90957b to 75f3f1c Compare December 11, 2022 20:35
neverchanje
neverchanje reviewed Dec 12, 2022
View reviewed changes
@waruto210 waruto210 force-pushed the waruto/s3-source branch 2 times, most recently from 245a813 to f88ea7a Compare December 12, 2022 12:03
@lmatz lmatz added the user-facing-changes Contains changes that are visible to users label Dec 12, 2022
@codecov
Copy link

codecov bot commented Dec 14, 2022 •
edited
Loading

Codecov Report

Merging #6846 (69e73c0) into main (8ef9c2a) will decrease coverage by 0.04%.
The diff coverage is 10.12%.

@@            Coverage Diff             @@
##             main    #6846      +/-   ##
==========================================
- Coverage   72.94%   72.89%   -0.05%     
==========================================
  Files        1038     1040       +2     
  Lines      167018   166969      -49     
==========================================
- Hits       121826   121713     -113     
- Misses      45192    45256      +64
Flag Coverage Δ
rust 72.89% <10.12%> (-0.05%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/batch/src/executor/source.rs 0.00% <0.00%> (ø)
src/connector/src/source/base.rs 66.13% <0.00%> (-9.17%) ⬇️
src/connector/src/source/filesystem/file_common.rs 0.00% <0.00%> (-80.36%) ⬇️
...c/connector/src/source/filesystem/s3/enumerator.rs 0.00% <0.00%> (ø)
src/connector/src/source/filesystem/s3/mod.rs 0.00% <0.00%> (ø)
...onnector/src/source/filesystem/s3/source/reader.rs 0.00% <0.00%> (ø)
src/source/src/connector_source.rs 76.89% <0.00%> (-0.33%) ⬇️
src/source/src/fs_connector_source.rs 0.00% <0.00%> (ø)
src/source/src/lib.rs 81.81% <0.00%> (-8.19%) ⬇️
src/source/src/manager.rs 71.30% <0.00%> (-7.47%) ⬇️
... and 16 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@waruto210 waruto210 force-pushed the waruto/s3-source branch 2 times, most recently from c64ef18 to 112a3e0 Compare December 15, 2022 04:41
@waruto210 waruto210 marked this pull request as ready for review December 15, 2022 04:41
@waruto210 waruto210 force-pushed the waruto/s3-source branch 2 times, most recently from 074d948 to 431884f Compare December 16, 2022 06:14
yezizp2012
yezizp2012 reviewed Dec 19, 2022
View reviewed changes
tabVersion
tabVersion approved these changes Dec 20, 2022
View reviewed changes
Copy link
Contributor

@tabVersion tabVersion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

generally LGTM

@waruto210
add s3 source and csv parser
d7482fc 
fix scan all completed split, add csv props

make some improvements
@mergify mergify bot merged commit f8782ba into main Dec 21, 2022
@mergify mergify bot deleted the waruto/s3-source branch December 21, 2022 06:38
@tabVersion tabVersion mentioned this pull request Dec 23, 2022
Closed
6 tasks
@hengm3467 hengm3467 mentioned this pull request Dec 27, 2022
Closed
@emile-00 emile-00 mentioned this pull request Jan 20, 2023
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yezizp2012 yezizp2012 yezizp2012 left review comments

@tabVersion tabVersion tabVersion approved these changes

@wangrunji0408 wangrunji0408 Awaiting requested review from wangrunji0408

@BugenZhao BugenZhao Awaiting requested review from BugenZhao

@shanicky shanicky Awaiting requested review from shanicky

+1 more reviewer

@neverchanje neverchanje neverchanje left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

type/feature Type: New feature. user-facing-changes Contains changes that are visible to users

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@waruto210 @neverchanje @tabVersion @yezizp2012 @lmatz