OCPNODE-3316: Track conmon process #9205

bitoku · 2025-05-15T17:15:40Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR adds feature to track conmon and logs error and emits a metric when the conmon is stopped.

Which issue(s) this PR fixes:

Fixes https://issues.redhat.com/browse/OCPNODE-1853

Special notes for your reviewer:

benchmark result is here.

Does this PR introduce a user-facing change?

Added feature to track conmon processes and emit containers_stopped_monitor_count{name="$ctr_name"} metric when it's stopped.

openshift-ci · 2025-05-15T17:15:45Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

bitoku · 2025-05-15T17:15:47Z

/test all

codecov · 2025-05-19T17:21:56Z

Codecov Report

❌ Patch coverage is 75.23810% with 26 lines in your changes missing coverage. Please review.
✅ Project coverage is 66.99%. Comparing base (e738af1) to head (1d392a1).
⚠️ Report is 136 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9205      +/-   ##
==========================================
+ Coverage   66.55%   66.99%   +0.44%     
==========================================
  Files         198      198              
  Lines       27164    27298     +134     
==========================================
+ Hits        18078    18288     +210     
+ Misses       7600     7505      -95     
- Partials     1486     1505      +19

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Signed-off-by: Ayato Tokubi <[email protected]>

bitoku · 2025-05-20T17:40:30Z

/skip

bitoku · 2025-05-21T08:13:38Z

/retest-required

Signed-off-by: Ayato Tokubi <[email protected]>

bitoku · 2025-06-10T06:05:20Z

@cri-o/cri-o-maintainers PTAL

openshift-ci-robot · 2025-06-12T05:49:13Z

@bitoku: This pull request references OCPNODE-3316 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.20.0" version, but no target version was set.

In response to this:

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR adds feature to track conmon and logs error and emits a metric when the conmon is stopped.

Which issue(s) this PR fixes:

Fixes https://issues.redhat.com/browse/OCPNODE-1853

Special notes for your reviewer:

benchmark result is here.

Does this PR introduce a user-facing change?
Added feature to track conmon processes and emit containers_stopped_monitor_count{name="$ctr_name"} metric when it's stopped. 

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

bitoku · 2025-06-18T02:00:33Z

@cri-o/cri-o-maintainers Can you PTAL?

bitoku · 2025-06-24T05:23:08Z

@cri-o/cri-o-maintainers Can you PTAL?

internal/lib/container_server.go

Signed-off-by: Ayato Tokubi <[email protected]>

haircommander · 2025-06-24T14:49:15Z

internal/oci/runtime_oci.go

 		return err
 	}

+	c.state.ContainerMonitorProcess, err = r.getConmonProcess(c)


could this be done in SetMonitorProcess?

I did it intentionally to make SetMonitorProcess reusable with conmon-rs.

internal/oci/runtime_oci.go

haircommander · 2025-06-24T14:56:14Z

internal/oci/runtime_pod.go

 	return c.Living() == nil
 }
+
+func (r *runtimePod) ProbeMonitor(ctx context.Context, c *Container) error {


we do save the PID of the monitor by asking it via RPC, we could do a similar thing as runtime oci here if we set the monitor pid to that

Yeah, but I'd prefer to do it in a separate PR and keep this PR small. (it's not small already though)

lance5890 · 2025-06-25T01:50:59Z

M

lance5890 · 2025-06-25T02:06:58Z

internal/oci/runtime_oci.go

+
+	if r.IsContainerAlive(c) {
+		metrics.Instance().MetricContainersStoppedMonitorCountInc(c.Name())
+		log.Errorf(ctx, "Conmon for container %s is stopped, although the container is running", c.ID())


should we restart the conmon process or stop the container ?

I'm not sure if we can restart conmon.
I'd rather not stop the container, because I think users want to decide what to do to the orphaned containers.

maybe we can add one more check to see if the container pid of pidfile is alive or not, if the pid of pidfile is dead, should remove the container，the pidfile lies in the following dir

[root@node userdata]# pwd /var/run/containers/storage/overlay-containers/abb2086e148c22259006ba00dcc360b953edf1277fcbc43d62588d936de0940b/userdata [root@node userdata]# ls attach config.json conmon-pidfile ctl pidfile run winsz

I have once encounted into this situation

Thanks but IsContainerAlive already checks container liveness with container pid. I don't understand why we need to use pidfile.

Signed-off-by: Ayato Tokubi <[email protected]>

bitoku · 2025-06-27T06:54:56Z

@cri-o/cri-o-maintainers PTAL?

haircommander · 2025-06-27T17:46:18Z

/override ci/prow/ci-e2e-evented-pleg
/approve

LGTM, good work here!

@cri-o/cri-o-maintainers PTAL

openshift-ci · 2025-06-27T17:46:25Z

@haircommander: Overrode contexts on behalf of haircommander: ci/prow/ci-e2e-evented-pleg

In response to this:

/override ci/prow/ci-e2e-evented-pleg
/approve

LGTM, good work here!

@cri-o/cri-o-maintainers PTAL

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci · 2025-06-30T06:25:43Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bitoku, haircommander, saschagrunert

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [haircommander,saschagrunert]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

bitoku · 2025-06-30T09:07:54Z

/retest

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 15, 2025

openshift-ci bot added dco-signoff: yes Indicates the PR's author has DCO signed all their commits. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels May 15, 2025

bitoku force-pushed the conmon-monitor3 branch from 38a1c67 to 78503b5 Compare May 16, 2025 11:09

bitoku mentioned this pull request May 16, 2025

[WIP] Add mechanism to monitor conmon #9172

Closed

bitoku force-pushed the conmon-monitor3 branch from 78503b5 to 09f6fb9 Compare May 19, 2025 17:13

bitoku force-pushed the conmon-monitor3 branch 6 times, most recently from 3bc2b2b to eb23ec3 Compare May 20, 2025 15:36

openshift-ci bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels May 20, 2025

bitoku changed the title ~~[WIP] Conmon monitor3~~ [WIP] Track conmon process May 20, 2025

Implement basic functionality needed to monitor conmon

8449d73

Signed-off-by: Ayato Tokubi <[email protected]>

bitoku force-pushed the conmon-monitor3 branch from eb23ec3 to d5199bc Compare May 20, 2025 15:58

bitoku marked this pull request as ready for review May 20, 2025 16:03

bitoku requested review from fidencio and mrunalp as code owners May 20, 2025 16:03

openshift-ci bot requested review from hasan4791 and klihub May 20, 2025 16:03

bitoku force-pushed the conmon-monitor3 branch from d5199bc to 16676e2 Compare May 20, 2025 17:52

Add probing loop and integration test

7c127d3

Signed-off-by: Ayato Tokubi <[email protected]>

bitoku force-pushed the conmon-monitor3 branch from 16676e2 to 7c127d3 Compare May 21, 2025 09:57

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 10, 2025

bitoku changed the title ~~Track conmon process~~ OCPNODE-3316: Track conmon process Jun 12, 2025

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jun 12, 2025

saschagrunert reviewed Jun 24, 2025

View reviewed changes

internal/lib/container_server.go Outdated Show resolved Hide resolved

accept context in probeMonitorProcesses

07e9b97

Signed-off-by: Ayato Tokubi <[email protected]>

haircommander reviewed Jun 24, 2025

View reviewed changes

internal/oci/runtime_oci.go Show resolved Hide resolved

haircommander reviewed Jun 24, 2025

View reviewed changes

internal/oci/runtime_oci.go Outdated Show resolved Hide resolved

haircommander reviewed Jun 24, 2025

View reviewed changes

haircommander mentioned this pull request Jun 24, 2025

container remains running when Process pid is died #9274

Closed

lance5890 reviewed Jun 25, 2025

View reviewed changes

Fix some comments and refactor in probeMonitor

1d392a1

Signed-off-by: Ayato Tokubi <[email protected]>

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 27, 2025

saschagrunert approved these changes Jun 30, 2025

View reviewed changes

openshift-ci bot assigned saschagrunert Jun 30, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 30, 2025

openshift-merge-bot bot merged commit f3374ad into cri-o:main Jun 30, 2025
106 of 108 checks passed

saschagrunert mentioned this pull request Jul 11, 2025

Implement ProbeMonitor for pod runtime monitor (conmon-rs) #9336

Closed

saschagrunert added the kind/feature Categorizes issue or PR as related to a new feature. label Aug 5, 2025

github-actions bot mentioned this pull request Sep 14, 2025

Bump crio from v1.30.3 to v1.34.0 kokyhm/kubespray#293

Open

OCPNODE-3316: Track conmon process #9205

OCPNODE-3316: Track conmon process #9205

Uh oh!

Conversation

bitoku commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

openshift-ci bot commented May 15, 2025

Uh oh!

bitoku commented May 15, 2025

Uh oh!

codecov bot commented May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

bitoku commented May 20, 2025

Uh oh!

bitoku commented May 21, 2025

Uh oh!

bitoku commented Jun 10, 2025

Uh oh!

openshift-ci-robot commented Jun 12, 2025

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

bitoku commented Jun 18, 2025

Uh oh!

bitoku commented Jun 24, 2025

Uh oh!

Uh oh!

haircommander Jun 24, 2025

Choose a reason for hiding this comment

Uh oh!

bitoku Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

haircommander Jun 24, 2025

Choose a reason for hiding this comment

Uh oh!

bitoku Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

lance5890 commented Jun 25, 2025

Uh oh!

lance5890 Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

bitoku Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

lance5890 Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

bitoku Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

bitoku commented Jun 27, 2025

Uh oh!

haircommander commented Jun 27, 2025

Uh oh!

openshift-ci bot commented Jun 27, 2025

Uh oh!

openshift-ci bot commented Jun 30, 2025

Uh oh!

bitoku commented Jun 30, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

bitoku commented May 15, 2025 •

edited

Loading

codecov bot commented May 19, 2025 •

edited

Loading

bitoku Jun 25, 2025 •

edited

Loading