Possible data race in kube.New because of AddToScheme #11169
Description
Issue
I'm using helm pkg in combination with controller-runtime to programmatically install Helm charts by Kubernetes operator.
Enabling the race detector prints several reports for unguarded concurrent access to the global scheme object.
It looks like #6566 fixes the problem between several kube.New concurrent invocations, however, it doesn't take into account that a global scheme object is used.
WARNING: DATA RACE
Write at 0x00c0003284e0 by goroutine 72:
runtime.mapassign()
/usr/local/Cellar/go/1.18.1/libexec/src/runtime/map.go:578 +0x0
k8s.io/apimachinery/pkg/runtime.(*Scheme).AddKnownTypeWithName()
/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/scheme.go:174 +0x3a4
k8s.io/apimachinery/pkg/runtime.(*Scheme).AddKnownTypes()
/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/scheme.go:148 +0x2e8
k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1.addKnownTypes()
/go/pkg/mod/k8s.io/[email protected]/pkg/apis/apiextensions/v1/register.go:48 +0x1d8
k8s.io/apimachinery/pkg/runtime.(*SchemeBuilder).AddToScheme()
/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/scheme_builder.go:29 +0x95
k8s.io/apimachinery/pkg/runtime.(*SchemeBuilder).AddToScheme-fm()
<autogenerated>:1 +0x44
helm.sh/helm/v3/pkg/kube.New.func1()
/go/pkg/mod/helm.sh/helm/[email protected]/pkg/kube/client.go:83 +0x59
sync.(*Once).doSlow()
/usr/local/Cellar/go/1.18.1/libexec/src/sync/once.go:68 +0x101
sync.(*Once).Do()
/usr/local/Cellar/go/1.18.1/libexec/src/sync/once.go:59 +0x46
helm.sh/helm/v3/pkg/kube.New()
/go/pkg/mod/helm.sh/helm/[email protected]/pkg/kube/client.go:82 +0x6a
helm.sh/helm/v3/pkg/action.(*Configuration).Init()
...{omitted}...
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114 +0x3ad
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler()
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311 +0x44a
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem()
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266 +0x324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227 +0xc8
Previous read at 0x00c0003284e0 by goroutine 104:
runtime.mapaccess2()
/usr/local/Cellar/go/1.18.1/libexec/src/runtime/map.go:456 +0x0
k8s.io/apimachinery/pkg/runtime.(*Scheme).New()
/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/scheme.go:296 +0xc8
k8s.io/apimachinery/pkg/runtime.UseOrCreateObject()
/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/codec.go:97 +0x149
k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode()
/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/serializer/json/json.go:188 +0xb93
k8s.io/apimachinery/pkg/runtime.WithoutVersionDecoder.Decode()
/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/helper.go:252 +0xb9
k8s.io/apimachinery/pkg/runtime.(*WithoutVersionDecoder).Decode()
<autogenerated>:1 +0xb5
sigs.k8s.io/controller-runtime/pkg/client/apiutil.targetZeroingDecoder.Decode()
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/client/apiutil/apimachinery.go:186 +0xd8
sigs.k8s.io/controller-runtime/pkg/client/apiutil.(*targetZeroingDecoder).Decode()
<autogenerated>:1 +0xb5
k8s.io/apimachinery/pkg/runtime.Decode()
/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/codec.go:58 +0x1f4
k8s.io/client-go/rest/watch.(*Decoder).Decode()
/go/pkg/mod/k8s.io/[email protected]/rest/watch/decoder.go:62 +0x18c
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive()
/go/pkg/mod/k8s.io/[email protected]/pkg/watch/streamwatcher.go:105 +0x19a
k8s.io/apimachinery/pkg/watch.NewStreamWatcher.func1()
/go/pkg/mod/k8s.io/[email protected]/pkg/watch/streamwatcher.go:76 +0x39
==================
==================
WARNING: DATA RACE
Write at 0x00c000328540 by goroutine 72:
runtime.mapassign()
/usr/local/Cellar/go/1.18.1/libexec/src/runtime/map.go:578 +0x0
k8s.io/apimachinery/pkg/runtime.(*Scheme).AddUnversionedTypes()
/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/scheme.go:128 +0x484
k8s.io/apimachinery/pkg/apis/meta/v1.AddToGroupVersion()
/go/pkg/mod/k8s.io/[email protected]/pkg/apis/meta/v1/register.go:75 +0x60a
k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1.addKnownTypes()
/go/pkg/mod/k8s.io/[email protected]/pkg/apis/apiextensions/v1/register.go:53 +0x20f
k8s.io/apimachinery/pkg/runtime.(*SchemeBuilder).AddToScheme()
/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/scheme_builder.go:29 +0x95
k8s.io/apimachinery/pkg/runtime.(*SchemeBuilder).AddToScheme-fm()
<autogenerated>:1 +0x44
helm.sh/helm/v3/pkg/kube.New.func1()
/go/pkg/mod/helm.sh/helm/[email protected]/pkg/kube/client.go:83 +0x59
sync.(*Once).doSlow()
/usr/local/Cellar/go/1.18.1/libexec/src/sync/once.go:68 +0x101
sync.(*Once).Do()
/usr/local/Cellar/go/1.18.1/libexec/src/sync/once.go:59 +0x46
helm.sh/helm/v3/pkg/kube.New()
/go/pkg/mod/helm.sh/helm/[email protected]/pkg/kube/client.go:82 +0x6a
helm.sh/helm/v3/pkg/action.(*Configuration).Init()
...{omitted}...
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114 +0x3ad
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler()
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311 +0x44a
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem()
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266 +0x324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227 +0xc8
Previous read at 0x00c000328540 by goroutine 39:
runtime.mapaccess2()
/usr/local/Cellar/go/1.18.1/libexec/src/runtime/map.go:456 +0x0
k8s.io/apimachinery/pkg/runtime.(*Scheme).ObjectKinds()
/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/scheme.go:267 +0x2f6
k8s.io/apimachinery/pkg/runtime.WithVersionEncoder.Encode()
/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/helper.go:223 +0x9e
k8s.io/apimachinery/pkg/runtime.(*WithVersionEncoder).Encode()
<autogenerated>:1 +0xfb
k8s.io/apimachinery/pkg/runtime.Encode()
/go/pkg/mod/k8s.io/[email protected]/pkg/runtime/codec.go:50 +0xb3
k8s.io/client-go/rest.(*Request).Body()
/go/pkg/mod/k8s.io/[email protected]/rest/request.go:450 +0x87c
k8s.io/client-go/kubernetes/typed/core/v1.(*events).CreateWithEventNamespace()
/go/pkg/mod/k8s.io/[email protected]/kubernetes/typed/core/v1/event_expansion.go:58 +0x22d
k8s.io/client-go/kubernetes/typed/core/v1.(*EventSinkImpl).Create()
/go/pkg/mod/k8s.io/[email protected]/kubernetes/typed/core/v1/event_expansion.go:160 +0x4a
k8s.io/client-go/tools/record.recordEvent()
/go/pkg/mod/k8s.io/[email protected]/tools/record/event.go:248 +0x1e1
k8s.io/client-go/tools/record.recordToSink()
/go/pkg/mod/k8s.io/[email protected]/tools/record/event.go:216 +0x237
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartRecordingToSink.func1()
/go/pkg/mod/k8s.io/[email protected]/tools/record/event.go:194 +0x7c
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
/go/pkg/mod/k8s.io/[email protected]/tools/record/event.go:311 +0xed
Would it be sufficient to create a new schema using runtime.NewScheme() and use it internally in the client instead of modifying the global one?
Output of helm version:v3.8.1
Output of kubectl version: v1.21.3
Cloud Provider/Platform: Minikube v1.25.2
Related to PR: #6566
Workaround
There are two workarounds:
- Create the Helm
action.Configurationand initialize it before starting the controller-runtime manager - Create a global dummy client and assign it to a blank identifier so that the
scheme.Schemeobject modification happens before any other code. E.g.var _ = kube.New(nil).