Skip to content

Conversation

mcshooter
Copy link
Contributor

@mcshooter mcshooter commented May 4, 2021

There have been some customer complaints that on windows node start ups, that when kube-proxy fails to start, some pods pods were entering into a crashloop state as services were inaccessible. Adding functionality to detect when kube-proxy is down, can attempt to repair the service and prevent pods from entering into a crashloop state.

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label May 4, 2021
@k8s-ci-robot
Copy link
Contributor

Hi @mcshooter. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels May 4, 2021
@mcshooter
Copy link
Contributor Author

/cc @jeremyje

@k8s-ci-robot k8s-ci-robot requested a review from jeremyje May 4, 2021 22:38
@jeremyje
Copy link
Contributor

jeremyje commented May 4, 2021

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 4, 2021
}

// healthCheckEndpointOKFunc returns a function to check the status of an http endpoint
func healthCheckEndpointOKFunc(endpoint string, timeout time.Duration) func() (bool, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like a missed opportunity to not make this generic health check through config. IE remove the hardcoded addresses and pass that in as a json argument.

/cc @Random-Liu what do you think?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Random-Liu what do you think?

Yeah, we thought about introducing the Kubernetes like "healthiness probe": Exec probe + http probe.

The only problem is that the health checker is already a plugin of NPD, and now we need another layer of plugin. :P

Some design is needed here.

Please feel free to suggest ideas or propose a design for this, but that doesn't necessarily need to block this change.

@mcshooter mcshooter force-pushed the detectKubeProxyServiceProblem branch from 1b79ad9 to 01cd8dd Compare May 5, 2021 17:28
@mcshooter
Copy link
Contributor Author

/retest

@jingxu97
Copy link
Contributor

jingxu97 commented May 5, 2021

@mcshooter this seems can be extended to support other windows services like csi-proxy which is installed in the same place?

cc @ddebroy

@mcshooter
Copy link
Contributor Author

@mcshooter this seems can be extended to support other windows services like csi-proxy which is installed in the same place?

cc @ddebroy

@jingxu97 I am not too familiar with what csi-proxy is, but if it's similar to kubelet and kube-proxy and having csi-proxy being down would cause the node to become unhealthy, then yes, it should be something that NPD should be able to support.

Copy link
Member

@Random-Liu Random-Liu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

}

// healthCheckEndpointOKFunc returns a function to check the status of an http endpoint
func healthCheckEndpointOKFunc(endpoint string, timeout time.Duration) func() (bool, error) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Random-Liu what do you think?

Yeah, we thought about introducing the Kubernetes like "healthiness probe": Exec probe + http probe.

The only problem is that the health checker is already a plugin of NPD, and now we need another layer of plugin. :P

Some design is needed here.

Please feel free to suggest ideas or propose a design for this, but that doesn't necessarily need to block this change.

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 5, 2021
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mcshooter, Random-Liu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 5, 2021
@k8s-ci-robot k8s-ci-robot merged commit 04e2b5f into kubernetes:master May 5, 2021
@mcshooter mcshooter deleted the detectKubeProxyServiceProblem branch May 5, 2021 23:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants