Summary of Changes

Hello @dentiny, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a new batch ingestion capability for the mooncake_table, allowing for efficient loading of data directly from Parquet files. This enhancement streamlines the process of populating tables with pre-existing data, complementing the existing streaming transaction mechanisms. It involves adding a new module for batch ingestion, integrating it into the table event handling system, and updating dependencies and test infrastructure to support this new functionality.

Highlights

• Batch Ingestion Functionality: A new asynchronous function, batch_ingest, has been added to the MooncakeTable struct. This function is responsible for efficiently loading data from a list of specified Parquet files into the table.
• New Table Event for Bulk Loading: A new event type, TableEvent::LoadFiles, has been introduced. This event carries the paths to Parquet files and a Logical Sequence Number (LSN), allowing external systems to trigger the batch ingestion process.
• Integration with Table Event Handler: The TableHandler has been updated to recognize and process the TableEvent::LoadFiles. Upon receiving this event, it dispatches the files to the MooncakeTable's new batch_ingest method.
• Dependency Addition (bytes crate): The bytes crate has been added as a dependency across relevant Cargo.toml files. This crate provides efficient byte buffers, which are crucial for handling the content of Parquet files during the ingestion process.
• Enhanced Testing Utilities and New Test Case: New test utilities, bulk_upload_files and generate_parquet_file, have been added to simplify the creation of test scenarios for batch ingestion. A dedicated test case, test_batch_ingestion, has also been implemented to validate the end-to-end bulk loading process.
• Transaction Stream Modifications: The commit_lsn field within TransactionStreamState has been made publicly accessible within the crate (pub(crate)), and a new constructor from_disk_files has been added to TransactionStreamCommit to support the new ingestion workflow.