Image pulls get stuck intermittently #6574
Description
What happened?
We host a private docker registry v 2.7.1-1 to host the images in our system. When a new pod was created the pod was assigned to a node and successfully created the pod network sandbox. The next event appearing in kubectl describe pod says pulling image, but that is the last event in the pod's event history with a Age of 50m.
Crictl pods shows the pod exists on the node it was scheduled to. I used ip netns exec to ensure the pod's network namespace was created appropriately and I was able to ping other IPs in the pod CIDR. Crictl ps does not show any containers that belong to the pod in question.
We replaced crio with containerd and the images pulled fine with both sequential and parallel pull settings in the kubelet.
What did you expect to happen?
For the image pull to indicate an error or success.
How can we reproduce it (as minimally and precisely as possible)?
This seems tough to do, but create a cluster with similar versions and private registry
Anything else we need to know?
No response
CRI-O and Kubernetes version
$ crio --version
Version: 1.19.1
GitCommit: 4e159cc8db813fb38e7429ba39ad2cf73b174446
GitTreeState: clean
BuildDate: 2021-01-21T00:18:46Z
GoVersion: go1.14.12
Compiler: gc
Platform: linux/amd64
Linkmode: dynamic$ kubectl --version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.4", GitCommit:"d360454c9bcd1634cf4cc52d1867af5491dc9c5f", GitTreeState:"clean", BuildDate:"2020-11-11T13:17:17Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.4", GitCommit:"d360454c9bcd1634cf4cc52d1867af5491dc9c5f", GitTreeState:"clean", BuildDate:"2020-11-11T13:09:17Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}OS version
# On Linux:
$ cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.7 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.7"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.7 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.7
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.7"
$ uname -a
Linux m2cisdp2 4.18.0-147.5.1.el8_1.x86_64 containers/image#1 SMP Tue Jan 14 15:50:19 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux