Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, Du…

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

Developer-friendly, serverless vector database for AI applications. Easily add long-term memory to your LLM apps!

A one-stop data processing system to make data higher-quality, juicier, and more digestible for LLMs! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷为大语言模型提供更高质量、更丰富、更易”消化“的数据！

Open, Multi-modal Catalog for Data & AI

🤗 The largest hub of ready-to-use datasets for ML models with fast, easy-to-use and efficient data manipulation tools

Minimalist ML framework for Rust

Rust client for the huggingface hub aiming for minimal subset of features over `huggingface-hub` python package

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

Upserts And Incremental Processing on Big Data

BitSail is a distributed, high-performance data integration framework and both support streaming and batch mode. At present, BitSail is mainly designed with the ELT model, which have EB data size a…

A native Rust library for Delta Lake, with bindings into Python

Apache Doris is an easy-to-use, high performance and unified analytics database.

RisingWave: the next-generation streaming database in the cloud.

Apache Flink

CDC Connectors for Apache Flink®

Change data capture for a variety of databases. Please log issues at https://issues.redhat.com/browse/DBZ.

Apache Pulsar - distributed pub-sub messaging system

This connector allows Apache Spark™ to read from and write to Delta Lake.

Airbyte is an open-source EL(T) platform that helps you replicate your data in your warehouses, lakes and databases.

Apache Arrow is a cross-language development platform for in-memory data. It specifies a standardized language-independent columnar memory format for flat and hierarchical data, organized for effic…

Apache Spark

An open source indexing subsystem that brings index-based query acceleration to Apache Spark™ and big data workloads.

The official home of the Presto distributed SQL query engine for big data

DataStax Spark Cassandra Connector

Apache Impala

Apache Parquet

Kafka Connect HDFS connector