Component(s)

receiver/filelog

What happened?

Description

Description:
I am using the filelog receiver in the OpenTelemetry Collector Contrib to parse CSV log files. When parsing a single file with a predefined header, the configuration works as expected. However, when attempting to process multiple CSV files with different headers, there is no way to dynamically handle varying headers.

If the header is omitted, the configuration fails with an error. This limitation makes it impossible to manage directories containing multiple CSV files with different structures efficiently.

Steps to Reproduce

Steps to Reproduce :

1. Configure the filelog receiver to parse a single CSV file with a specified header
   receivers:
   filelog/LightningInteractionLogs_quoted:
   include: [/u01/SFLogs/8292024/continuationcallout_hundred.csv]
   start_at: beginning
   operators:

   • type: csv_parser
     header: ApplicationName, page_app_name, Application_Version, Environment, HostName, EventType, timestamp, user_id, user_name, url, duration, request_form_size, response_size, status_code, success, TimestampDerived

2. Attempt to configure the receiver to include multiple CSV files with varying headers:
   receivers:
   filelog/LightningInteractionLogs_multiple:
   include: [/u01/SFLogs/*.csv]
   start_at: beginning
   operators:

   • type: csv_parser

     No way to handle multiple headers dynamically

3. Observe the failure when the header is not explicitly provided:
   Error: failed to build pipelines: failed to create "filelog/LightningInteractionLogs_multiple" receiver for data type "logs"; missing required field "header" or "header_attribute"

Expected Result

Expected Result :
The csv_parser operator should be able to:

Dynamically detect headers from the first row of the CSV file (e.g., via a dynamic_header option).
Alternatively, allow mapping specific headers to specific files or file patterns using a header_attribute or similar configuration.

For example:
receivers:
filelog/LightningInteractionLogs_dynamic:
include: [/u01/SFLogs/*.csv]
start_at: beginning
operators:
- type: csv_parser
dynamic_header: true

Actual Result

Actual Result
The configuration fails when header is not explicitly provided, making it impossible to process multiple CSV files with different headers in the same receiver configuration.

Error message:
Error: failed to build pipelines: failed to create "filelog/LightningInteractionLogs_multiple" receiver for data type "logs"; missing required field "header" or "header_attribute"

Collector version

v0.109.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

receivers:
  filelog/LightningInteractionLogs_multiple:
    include: [/u01/SFLogs/*.csv]
    start_at: beginning
    operators:
      - type: csv_parser
exporters:
  logging:
    loglevel: debug

service:
  pipelines:
    logs:
      receivers: [filelog/LightningInteractionLogs_multiple]
      exporters: [logging]

Log output

Error: failed to build pipelines: failed to create "filelog/LightningInteractionLogs_multiple" receiver for data type "logs"; missing required field "header" or "header_attribute"

Additional context

No response