• Overview
• Key components
  • Log ingestion and storage
  • Query Language (ferum-ql or fql).
    • Key Design Features:
  • Servers
    • OpenTelemetry gRPC server
    • Internal gRPC server
    • HTTP API server
  • Configuration management
• Performance Considerations
• Deployment and Runtime
• Roadmap
  • In progress
  • Planned (short term)
  • Planned (long term)

Ferrum is a sophisticated log management system built in Rust, designed to handle log ingestion from opentelemetry, storage, querying, and processing using modern data processing techniques. The project consists of two main Rust crates: ferum-ql (a query language parser) and logs (the main log management application).

• Custom query language (see FQL and the HTTP API)
• SQL query support (see HTTP API)
• Replication with WAL and multi-node support (via Raft)
• File compaction (configurable frequency)
• OpenTelemetry logs compatible gRPC server (see OpenTelemetry gRPC server)
• HTTP for querying data (see HTTP API)
• Object storage support (currently only AWS S3)

The logs system is built around several components:

1. Data model. Logs are stored with a rich schema including

   • Timestamp (nanosecond precision)
   • Log level
   • Message
   • Attributes (as a map)
   • Day partitioning

2. Table management

   • Uses Apache Arrow and DataFusion for high-performance data processing
   • Implements compaction and periodic file management
   • Enables efficient querying and storage of log data using parquet

3. Filesystems

   • Local
   • S3

Located in ferum-ql crate, this is a custom query language parser build with LALRPOP . It allows users to:

• Filter logs by level, message, and custom attributes
• Apply operations like equality, inequality, and regex matching
• Perform functions like counting and JSON transformation (message field)

Example query syntax: {level="Error", message~="connection.*", myattribute="something"} | count

This query would:

1. Filter logs with Error level
2. Match messages with a "connection" regex pattern
3. Count the results

• Strongly typed parsing
• Supports multiple filter operations
• Flexible attribute filtering
• Function extensions (count, JSON conversion for message fields)

The application provides three server interfaces:

1. Implements the OpenTelemetry LogsService
2. Receives log entries from distributed systems
3. Transforms incoming logs into the internal data model
4. High-performance log ingestion

Handles replication and internode communication

External API for querying data with the following endpoints:

1. /query/fql: Ferum Query Language endpoint
2. /query/sql: Direct SQL querying
3. /query/attributes: Query attribute keys based on FQL query
4. /query/attributes/{attribute}/values: Query attribute values for a specific attribute

The system supports flexible configuration via YAML:

• Data directory configuration
• Log table settings (e.g., compaction frequency)
• Server port configurations
• Replication settings

See example config for details.

• Uses jemallocator for memory management
• Leverages Apache Arrow for columnar data processing
• Implements zero-copy data transformations
• Supports parallel processing

• Supports graceful shutdown
• Configurable log levels
• Environment-aware configuration loading
• Handles system signals (SIGTERM, CTRL+C)

• S3 storage backend
• Improve compaction strategy for better disk usage
• Docker support

• Distributed queries on multi-node setups
• Refactor into multiple crates for better benchmarking and testing
• Add functions to FQL (sum, rate etc.)

• Kubernetes operator
• Data retention policies and TTL's
• Query caching for improved performance