Initial commit for setting up a new component: k8slog receiver #24439

h0cheung · 2023-07-24T08:05:18Z

Description:

k8slog receiver provides full functionality to collect logs in k8s containers. It is an all-in-one solution, which supports to:

collect logs from files inside k8s containers via daemonset
support docker and cri-containerd
support many docker graph drivers / snapshotters
collect logs from stdout of k8s containers via daemonset
filter containers by metadata
extract metadata of containers
collect file logs in k8s containers via sidecar

If we want to collect logs from k8s, we can use this component.

Link to tracking Issue:

#23339

Testing:

Tests for unmarshaling configuration have been added.

Documentation:

Docs for configurations and overall design has been added.

github-actions · 2023-08-08T05:20:07Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

h0cheung · 2023-08-08T13:34:58Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

remove stale

dmitryax · 2023-08-14T20:55:50Z

receiver/k8slogreceiver/config.go

Are all of the configuration options required for the initial implementation? Can you remove them for now and add them along with the functionality using them going forward?

I agree, starting small will make things easier to review and get started and we can always add them in the future.

TylerHelmuth

@h0cheung I am very excited about this component, sorry I didn't see your proposal sooner. @dmitryax thanks for pointing it out.

TylerHelmuth · 2023-08-14T21:22:59Z

receiver/k8slogreceiver/config.go

Lets not introduce expr filtering to start. The other filter options you've provided are quite extensive. For expression language filtering I'd prefer we use OTTL.

receiver/k8slogreceiver/config.go

TylerHelmuth · 2023-08-14T21:28:38Z

receiver/k8slogreceiver/metadata.yaml

You can add me here as well

OK, I will do it soon.

TylerHelmuth · 2023-08-14T21:29:42Z

receiver/k8slogreceiver/config.go

I agree, starting small will make things easier to review and get started and we can always add them in the future.

receiver/k8slogreceiver/config.go

h0cheung · 2023-09-04T02:42:50Z

@TylerHelmuth @dmitryax I've simplified the configs and added Tyler Helmuth to the code owner. Could you please have a look at this?

TylerHelmuth · 2023-09-06T22:35:27Z

receiver/k8slogreceiver/config.go

I don't think we should allow extracting from env variables, that feels like a security risk.

OtelEnv has been removed.
However, Env is needed in some cases. For example, if there are some environment variables in the docker image, there is no other way to get it.
Is this really risky? resourcedetection supports it, and this component gets k8s node name from environment variables, too.

Using an env var to filter feels ok. But this section is for taking fields from the pod and adding them as resource attributes to the log right?

Getting attributes from environment variables is useful, and many components support it such as resoucedetection

Are there any reports of security risks associated with reading environment variables as strings and adding them to data?

The risk is exposing apikey/password information, but that risk is lowered if the config only allows static strings. If we do allow this I think we can do all env vars with 1 config option, no need to split out OTel env vars.

By default, the env config is empty, and no variable will be exposed. When a user set this, he should know what he is doing.

It seems that otel_env is necessary, so I've removed it in the latest commit.

@dmitryax do you have an opinion?

Any updates for this?

Lets move forward with this allowed as we've discussed.

I also find it useful to have the option to collect ENV_VAR information from the Pods.
That would give us important information about OTEL specific info like OTEL_SERVICE_NAME.
Having service.name populated in log records collected by the k8slog receiver is quite important since it will allow us to correlate logs with traces using the service.name value.

receiver/k8slogreceiver/config.go

.github/CODEOWNERS

MovieStoreGuy · 2025-01-24T01:31:53Z

Hey @h0cheung,

Are you able to follow up and resolve any of the outstanding comments so we can move forward this PR?

atoulme · 2025-03-17T21:49:59Z

We need to address this PR. We have once again the need to update it to the latest dependencies so it will merge with main.

I propose that we wait after the next release of contrib and update one more time all the code to the latest main, and get this in. This PR is marked as never stale and needs to be addressed one way or another.

If this plan of action is impractical, I would move regrettably to offering that we push this PR. I don't think it will come to that.

Co-authored-by: Chris Mark <[email protected]>

h0cheung · 2025-03-18T08:48:06Z

We need to address this PR. We have once again the need to update it to the latest dependencies so it will merge with main.

I propose that we wait after the next release of contrib and update one more time all the code to the latest main, and get this in. This PR is marked as never stale and needs to be addressed one way or another.

If this plan of action is impractical, I would move regrettably to offering that we push this PR. I don't think it will come to that.

I saw that v0.122.0 was released and updated it. I think we can merge it soon after the pipeline is passed.

…telemetry#24439) **Description:** <Describe what has changed.>  k8slog receiver provides full functionality to collect logs in k8s containers. It is an all-in-one solution, which supports to: - collect logs from files inside k8s containers via daemonset - support docker and cri-containerd - support many docker graph drivers / snapshotters - collect logs from stdout of k8s containers via daemonset - filter containers by metadata - extract metadata of containers - collect file logs in k8s containers via sidecar If we want to collect logs from k8s, we can use this component. **Link to tracking Issue:** <Issue number if applicable> open-telemetry#23339 **Testing:** <Describe what testing was performed and which tests were added.> Tests for unmarshaling configuration have been added. **Documentation:** <Describe the documentation added.> Docs for configurations and overall design has been added. --------- Co-authored-by: Chris Mark <[email protected]>

h0cheung requested review from a team and kovrus July 24, 2023 08:05

github-actions bot assigned bogdandrutu Jul 24, 2023

h0cheung force-pushed the feat/k8slog-setup branch from 4858200 to 3e05fa1 Compare July 24, 2023 10:02

github-actions bot added the Stale label Aug 8, 2023

h0cheung force-pushed the feat/k8slog-setup branch from 3e05fa1 to ea3a841 Compare August 8, 2023 13:42

github-actions bot removed the Stale label Aug 9, 2023

h0cheung force-pushed the feat/k8slog-setup branch 6 times, most recently from 951c26c to 04d4b40 Compare August 10, 2023 06:39

dmitryax mentioned this pull request Aug 14, 2023

Improve Kubernetes Logs Collection Experience #25251

Closed

dmitryax reviewed Aug 14, 2023

View reviewed changes

TylerHelmuth reviewed Aug 14, 2023

View reviewed changes

TylerHelmuth removed the request for review from kovrus August 14, 2023 21:32

dmitryax mentioned this pull request Aug 28, 2023

New component: Kubernetes api logs receiver #24641

Closed

h0cheung force-pushed the feat/k8slog-setup branch from 04d4b40 to 6eb08af Compare September 1, 2023 12:39

h0cheung requested a review from atoulme as a code owner September 1, 2023 12:39

github-actions bot added the cmd/githubgen label Sep 1, 2023

h0cheung force-pushed the feat/k8slog-setup branch 2 times, most recently from 82afa36 to 5fde154 Compare September 1, 2023 13:06

TylerHelmuth reviewed Sep 6, 2023

View reviewed changes

h0cheung force-pushed the feat/k8slog-setup branch 3 times, most recently from 132b260 to 8256f32 Compare September 8, 2023 12:35

h0cheung force-pushed the feat/k8slog-setup branch 2 times, most recently from 09fca1f to cb65319 Compare September 19, 2024 12:05

h0cheung requested a review from a team as a code owner September 19, 2024 12:05

atoulme reviewed Sep 19, 2024

View reviewed changes

.github/CODEOWNERS Outdated Show resolved Hide resolved

h0cheung force-pushed the feat/k8slog-setup branch 2 times, most recently from c02d865 to 076dead Compare September 24, 2024 02:55

h0cheung force-pushed the feat/k8slog-setup branch 2 times, most recently from 2207485 to aea45ae Compare October 21, 2024 06:34

h0cheung force-pushed the feat/k8slog-setup branch 6 times, most recently from 00bdf01 to 03c0b71 Compare January 26, 2025 03:31

h0cheung force-pushed the feat/k8slog-setup branch from 03c0b71 to c580de4 Compare March 18, 2025 08:30

h0cheung and others added 6 commits March 18, 2025 16:34

feat: initial commit for k8slog receiver

9b148e0

mod: only try to connect to cri by default

65f1504

mod: update README to respect the code changes

e0c2d11

docs (receiver/k8slog): describe detailed design for each mode.

9b3fd6b

feat (receiver/k8slog): update dependencies.

c41de77

docs(receiver/k8slog): fix typo

ff48087

Co-authored-by: Chris Mark <[email protected]>

h0cheung force-pushed the feat/k8slog-setup branch from c580de4 to 4e7676a Compare March 18, 2025 08:34

(receiver/k8slog): sync dependencies.

581cf9f

h0cheung force-pushed the feat/k8slog-setup branch from 4e7676a to 581cf9f Compare March 18, 2025 09:07

atoulme merged commit 5d4cd9d into open-telemetry:main Mar 18, 2025
171 checks passed

github-actions bot added this to the next release milestone Mar 18, 2025

Initial commit for setting up a new component: k8slog receiver #24439

Initial commit for setting up a new component: k8slog receiver #24439

Uh oh!

Conversation

h0cheung commented Jul 24, 2023

Uh oh!

github-actions bot commented Aug 8, 2023

Uh oh!

h0cheung commented Aug 8, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TylerHelmuth left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

h0cheung Aug 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

h0cheung commented Sep 4, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

MovieStoreGuy commented Jan 24, 2025

Uh oh!

atoulme commented Mar 17, 2025

Uh oh!

h0cheung commented Mar 18, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

h0cheung Aug 20, 2023 •

edited

Loading