Skip to content

[fileconsumer] Add compression:auto to support ingesting mixed file types #38510

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 30 commits into from
May 9, 2025
Merged

[fileconsumer] Add compression:auto to support ingesting mixed file types #38510

merged 30 commits into from
May 9, 2025

Conversation

khushijain21
Copy link
Contributor

@khushijain21 khushijain21 commented Mar 11, 2025 •
edited
Loading

Description

This PR supports the use case of reading from mixed file types. To support reading from both compressed and uncompressed file - configure

compression: auto

Link to tracking issue

Fixes #37772

Testing

Start otelcollector-contrib with following config file

receivers:
  filelog:
    include: [ "./test.log*" ]  # Path to log file
    start_at: beginning              # Read from the start of the file
    include_file_path: true
    include_file_name: true
    compression: auto
  

exporters:
  debug/1:
    verbosity: detailed

service:
  telemetry:
    logs: 
      level: debug
  pipelines:
    logs:
      receivers: [filelog]
      exporters: [debug/1]

Have test.log and test.log.gz in your path. We have two readers - one for plain text and other for gzipped file.

@khushijain21 khushijain21 marked this pull request as draft March 11, 2025 05:55
@khushijain21 khushijain21 changed the title Support mixed compression use case [fileconsumer] Support mixed compression use case Mar 11, 2025
@khushijain21 khushijain21 marked this pull request as ready for review March 11, 2025 09:00
khushijain21
khushijain21 commented Mar 11, 2025
View reviewed changes
VihasMakwana
VihasMakwana reviewed Mar 17, 2025
View reviewed changes
VihasMakwana
VihasMakwana reviewed Mar 17, 2025
View reviewed changes
@atoulme atoulme marked this pull request as draft March 17, 2025 22:03
@khushijain21 khushijain21 marked this pull request as ready for review March 18, 2025 10:28
@khushijain21 khushijain21 marked this pull request as draft March 18, 2025 10:28
@khushijain21
Copy link
Contributor Author

@atoulme any reason you converted this to draft? 🤔

@github-actions github-actions bot assigned