0.25.2 operator hangs waiting on reconciliation for storage configmap #1781
Description
After an upgrade from 0.24.5 to 0.25.2. the Altinity operator hung waiting for a configmap (apparently) and did not complete. The expected behavior is that reconciliation would complete. Instead, the operator hung and kept printing the following messages. 'xxx' is a stand-in for the real CHI resource name.
Poll():clickhouse/chi-storage-logs-0:poll iteration
I0729 02:42:38.215739 1 poller.go:111] Poll():clickhouse/chi-storage-logs-0:WAIT: clickhouse/chi-storage-xxxxxx
I0729 02:42:38.215761 1 poller.go:115] Poll():clickhouse/chi-storage-logs-0:poll iteration
I0729 02:42:48.228778 1 poller.go:111] Poll():clickhouse/chi-storage-logs-0:WAIT: clickhouse/chi-storage-xxxxxx
I0729 02:42:48.228798 1 poller.go:115] Poll():clickhouse/chi-storage-logs-0:poll iteration
I0729 02:42:58.242363 1 poller.go:111] Poll():clickhouse/chi-storage-logs-0:WAIT: clickhouse/chi-storage-xxxxxx
I0729 02:42:58.242383 1 poller.go:115] Poll():clickhouse/chi-storage-logs-0:poll iteration
The CHI resource showed status InProgress, e.g.,
kubectl get chi
NAME CLUSTERS HOSTS STATUS HOSTS-COMPLETED AGE SUSPEND
xxxxxx 1 30 InProgress 30 98d
This does not resolve itself. If you encounter it, the workaround is to delete the configmap. At that point reconciliation completes normally and the operator is OK from then on.