[processor/resourcedetection] Azure provider fails with timeout >30s #40372
Description
Component(s)
internal/metadataproviders
What happened?
Description
When deployed as daemonset on Kubernetes, a collector configured with a package that uses the azure metadata provider (such as resourcedetection processor or datadog exporter) and a timeout context greater than 30 seconds is passed to azure.Provider.Metadata function, collector pod will continuously restart every 30 seconds.
Steps to Reproduce
- edit
./examples/kubernetes/daemonset-collector-dev.yamlto contain the following processor and add it to a pipeline:
processors:
resourcedetection:
timeout: 35s
detectors:
- azure
...
service:
pipelines:
metrics:
receivers: [kubeletstats]
processors: [resourcedetection, batch]
exporters: [debug]
- run the following commands:
make kind-build
make kind-install-daemonset
Expected Result
resource detection times out after 35 seconds or specified timeout value
Actual Result
pod restarts every 30 seconds
Collector version
main (v0.127.0+)
Environment information
Environment
OS: Linux, Kubernetes (kind)
Compiler(if manually compiled): (e.g., "go 14.2")
OpenTelemetry Collector configuration
mode: daemonset
image:
repository: otelcontribcol-dev
tag: "0.0.1"
pullPolicy: IfNotPresent
command:
name: otelcontribcol
extraEnvs:
- name: K8S_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
clusterRole:
create: true
rules:
- apiGroups:
- ''
resources:
- 'pods'
- 'nodes'
verbs:
- 'get'
- 'list'
- 'watch'
- apiGroups: [ "" ]
resources: [ "nodes/stats", "nodes/proxy"]
verbs: [ "get" ]
config:
exporters:
debug:
verbosity: detailed
receivers:
kubeletstats:
collection_interval: 10s
auth_type: 'serviceAccount'
endpoint: '${env:K8S_NODE_NAME}:10250'
insecure_skip_verify: true
k8s_api_config:
auth_type: serviceAccount
metrics:
container.cpu.utilization:
enabled: true
container.cpu.time:
enabled: true
container.cpu.usage:
enabled: true
k8s.container.cpu_limit_utilization:
enabled: true
service:
pipelines:
metrics:
receivers: [kubeletstats]
processors: [batch]
exporters: [debug]Log output
john.peterson@COMP-HHVKJYJXCY opentelemetry-collector-contrib % kubectl logs daemonset-collector-dev-opentelemetry-collector-agent-6vqjf
2025-05-29T17:30:15.689Z info [email protected]/service.go:199 Setting up own telemetry... {"resource": {}}
2025-05-29T17:30:15.689Z info builders/builders.go:26 Development component. May change in the future. {"resource": {}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics"}
2025-05-29T17:30:15.689Z debug builders/builders.go:24 Beta component. May change in the future. {"resource": {}, "otelcol.component.id": "batch", "otelcol.component.kind": "processor", "otelcol.pipeline.id": "metrics", "otelcol.signal": "metrics"}
2025-05-29T17:30:15.689Z debug builders/builders.go:24 Beta component. May change in the future. {"resource": {}, "otelcol.component.id": "memory_limiter", "otelcol.component.kind": "processor", "otelcol.pipeline.id": "metrics", "otelcol.signal": "metrics"}
2025-05-29T17:30:15.689Z debug [email protected]/factory.go:144 created singleton logger {"resource": {}, "otelcol.component.kind": "processor"}
2025-05-29T17:30:15.690Z info [email protected]/memorylimiter.go:149 Using percentage memory limiter {"resource": {}, "otelcol.component.kind": "processor", "total_memory_mib": 15972, "limit_percentage": 80, "spike_limit_percentage": 25}
2025-05-29T17:30:15.690Z info [email protected]/memorylimiter.go:74 Memory limiter configured {"resource": {}, "otelcol.component.kind": "processor", "limit_mib": 12777, "spike_limit_mib": 3993, "check_interval": 5}
2025-05-29T17:30:15.690Z info builders/builders.go:26 Development component. May change in the future. {"resource": {}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "traces"}
2025-05-29T17:30:15.690Z debug builders/builders.go:24 Beta component. May change in the future. {"resource": {}, "otelcol.component.id": "batch", "otelcol.component.kind": "processor", "otelcol.pipeline.id": "traces", "otelcol.signal": "traces"}
2025-05-29T17:30:15.690Z debug builders/builders.go:24 Stable component. {"resource": {}, "otelcol.component.id": "otlp", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics"}
2025-05-29T17:30:15.690Z debug [email protected]/otlp.go:58 created signal-agnostic logger {"resource": {}, "otelcol.component.id": "otlp", "otelcol.component.kind": "receiver"}
2025-05-29T17:30:15.690Z info builders/builders.go:26 Development component. May change in the future. {"resource": {}, "otelcol.component.id": "debug", "otelcol.component.kind": "exporter", "otelcol.signal": "logs"}
2025-05-29T17:30:15.690Z debug builders/builders.go:24 Beta component. May change in the future. {"resource": {}, "otelcol.component.id": "batch", "otelcol.component.kind": "processor", "otelcol.pipeline.id": "logs", "otelcol.signal": "logs"}
2025-05-29T17:30:15.690Z debug builders/builders.go:24 Beta component. May change in the future. {"resource": {}, "otelcol.component.id": "memory_limiter", "otelcol.component.kind": "processor", "otelcol.pipeline.id": "logs", "otelcol.signal": "logs"}
2025-05-29T17:30:15.690Z debug builders/builders.go:24 Stable component. {"resource": {}, "otelcol.component.id": "otlp", "otelcol.component.kind": "receiver", "otelcol.signal": "logs"}
2025-05-29T17:30:15.690Z debug builders/builders.go:24 Beta component. May change in the future. {"resource": {}, "otelcol.component.id": "resourcedetection", "otelcol.component.kind": "processor", "otelcol.pipeline.id": "traces", "otelcol.signal": "traces"}
2025-05-29T17:30:15.690Z debug builders/builders.go:24 Stable component. {"resource": {}, "otelcol.component.id": "otlp", "otelcol.component.kind": "receiver", "otelcol.signal": "traces"}
2025-05-29T17:30:15.690Z debug builders/builders.go:24 Beta component. May change in the future. {"resource": {}, "otelcol.component.id": "prometheus", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics"}
2025-05-29T17:30:15.690Z debug builders/extension.go:48 Beta component. May change in the future. {"resource": {}, "otelcol.component.id": "health_check", "otelcol.component.kind": "extension"}
2025-05-29T17:30:15.691Z info [email protected]/service.go:266 Starting otelcontribcol... {"resource": {}, "Version": "0.126.0-dev", "NumCPU": 16}
2025-05-29T17:30:15.691Z info extensions/extensions.go:41 Starting extensions... {"resource": {}}
2025-05-29T17:30:15.691Z info extensions/extensions.go:45 Extension is starting... {"resource": {}, "otelcol.component.id": "health_check", "otelcol.component.kind": "extension"}
2025-05-29T17:30:15.691Z info healthcheckextension/healthcheckextension.go:32 Starting health_check extension {"resource": {}, "otelcol.component.id": "health_check", "otelcol.component.kind": "extension", "config": {"Endpoint":"10.244.0.51:13133","TLSSetting":null,"CORS":null,"Auth":null,"MaxRequestBodySize":0,"IncludeMetadata":false,"ResponseHeaders":null,"CompressionAlgorithms":null,"ReadTimeout":0,"ReadHeaderTimeout":0,"WriteTimeout":0,"IdleTimeout":0,"Middlewares":null,"Path":"/","ResponseBody":null,"CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
2025-05-29T17:30:15.691Z info extensions/extensions.go:62 Extension started. {"resource": {}, "otelcol.component.id": "health_check", "otelcol.component.kind": "extension"}
2025-05-29T17:30:15.691Z info [email protected]/server.go:690 [core] [Server #1]Server created {"resource": {}, "grpc_log": true}
2025-05-29T17:30:15.691Z info [email protected]/otlp.go:116 Starting GRPC server {"resource": {}, "otelcol.component.id": "otlp", "otelcol.component.kind": "receiver", "endpoint": "0.0.0.0:4317"}
2025-05-29T17:30:15.692Z info [email protected]/otlp.go:173 Starting HTTP server {"resource": {}, "otelcol.component.id": "otlp", "otelcol.component.kind": "receiver", "endpoint": "0.0.0.0:4318"}
2025-05-29T17:30:15.692Z info internal/resourcedetection.go:137 began detecting resource information {"resource": {}, "otelcol.component.id": "resourcedetection", "otelcol.component.kind": "processor", "otelcol.pipeline.id": "traces", "otelcol.signal": "traces"}
2025-05-29T17:30:15.692Z info internal/resourcedetection.go:142 detecting resource {"resource": {}, "otelcol.component.id": "resourcedetection", "otelcol.component.kind": "processor", "otelcol.pipeline.id": "traces", "otelcol.signal": "traces", "detector": {}}
2025-05-29T17:30:15.692Z info internal/resourcedetection.go:144 running goroutine detector {"resource": {}, "otelcol.component.id": "resourcedetection", "otelcol.component.kind": "processor", "otelcol.pipeline.id": "traces", "otelcol.signal": "traces", "detector": {}}
2025-05-29T17:30:15.692Z debug azure/metadata.go:72 Getting metadata {"resource": {}, "otelcol.component.id": "resourcedetection", "otelcol.component.kind": "processor", "otelcol.pipeline.id": "traces", "otelcol.signal": "traces", "endpoint": "http://169.254.169.254/metadata/instance/compute"}
2025-05-29T17:30:15.692Z debug azure/metadata.go:76 Azure request context {"resource": {}, "otelcol.component.id": "resourcedetection", "otelcol.component.kind": "processor", "otelcol.pipeline.id": "traces", "otelcol.signal": "traces", "ctx": "context.Background.WithValue(internal.contextKey, *http.Client).WithDeadline(2025-05-29 17:30:50.692040961 +0000 UTC m=+35.090058168 [34.999724167s])"}
2025-05-29T17:30:15.692Z debug azure/metadata.go:78 Created request {"resource": {}, "otelcol.component.id": "resourcedetection", "otelcol.component.kind": "processor", "otelcol.pipeline.id": "traces", "otelcol.signal": "traces"}
2025-05-29T17:30:15.692Z debug azure/metadata.go:89 Encoded query {"resource": {}, "otelcol.component.id": "resourcedetection", "otelcol.component.kind": "processor", "otelcol.pipeline.id": "traces", "otelcol.signal": "traces", "req": "http://169.254.169.254/metadata/instance/compute?api-version=2020-09-01&format=json"}
2025-05-29T17:30:15.692Z info [email protected]/server.go:886 [core] [Server #1 ListenSocket #2]ListenSocket created {"resource": {}, "grpc_log": true}
2025-05-29T17:30:20.691Z debug [email protected]/memorylimiter.go:179 Currently used memory. {"resource": {}, "otelcol.component.kind": "processor", "cur_mem_mib": 24}
2025-05-29T17:30:25.690Z debug [email protected]/memorylimiter.go:179 Currently used memory. {"resource": {}, "otelcol.component.kind": "processor", "cur_mem_mib": 24}
2025-05-29T17:30:30.690Z debug [email protected]/memorylimiter.go:179 Currently used memory. {"resource": {}, "otelcol.component.kind": "processor", "cur_mem_mib": 24}
2025-05-29T17:30:35.690Z debug [email protected]/memorylimiter.go:179 Currently used memory. {"resource": {}, "otelcol.component.kind": "processor", "cur_mem_mib": 24}Additional context
Not sure if it makes sense to just set default context when creating azure provider, or if this is a deeper issue with hanging http requests on k8s