Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.

Apache Spark

High performance distributed cache system. Built by Rust.

🚀 High-performance distributed object storage for MinIO alternative.

Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, Du…

A lightweight, powerful framework for multi-agent workflows

A high-throughput and memory-efficient inference and serving engine for LLMs

User tools for Spark RAPIDS

A high-performance distributed file system designed to address the challenges of AI training and inference workloads.

A Cloud Native Batch System (Project under CNCF)

Apache Hadoop

DAOS Storage Stack (client libraries, storage engine, control plane)

Storage Orchestration for Kubernetes

A QoS-based scheduling system brings optimal layout and status to workloads such as microservices, web services, big data jobs, AI jobs, etc.

Fluss is a streaming storage built for real-time analytics.

Storage Performance Development Kit

LBA tools(hd_write_verify & hd_write_verify_dump) are very useful for testing Storage stability and verifying DATA consistency, there are much better than FIO & vdbench's verifying functions. for e…

Hive federation service. Enables disparate tables to be concurrently accessed across multiple Hive deployments.

Apache Hive

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.

CSI driver for Ceph

HiBench is a big data benchmark suite.

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

Python SQL Parser and Transpiler

A curated list of awesome StarkNet resources, libraries, tools and more

LiteIO is a cloud-native block device service that uses multiple storage engines, including SPDK and LVM, to achieve high performance. It is specifically designed for Kubernetes in a hyper-converge…

Spark RAPIDS plugin - accelerate Apache Spark with GPUs