datadog receiver causing collector to panic/crash with invalid memory address or nil pointer deference #40557
Description
Component(s)
receiver/datadog
What happened?
Description
We I enabled the datadog receiver and started to send telemetry to it, my collector will panic and restart continuously (daemonset deployment via operator).
http: panic serving 10.39.3.1:54054: runtime error: invalid memory address or nil pointer dereference
Steps to Reproduce
- Configure the OTel Collector to use the datadog receiver and create a simple pipeline for handling traces.
- Enable the
receiver.datadogreceiver.Enable128BitTraceIDfeature gate
receivers:
datadog:
endpoint: '${env:K8S_POD_IP}:8126'
read_timeout: 60s
# trace_id_cache_size: 1000 // making the cache size larger makes the crash happen after a longer amount of time, but it still happens
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 80
spike_limit_percentage: 20
batch:
send_batch_max_size: 1100
send_batch_size: 1000
timeout: 1s
exporters:
debug:
verbosity: detailed
pipelines:
traces:
exporters:
- debug
processors:
- memory_limiter
- batch
receivers:
- datadogExpected Result
Telemetry received and processed without errors/crashing
Actual Result
Collector will continuously panic/crash and restart (kubernetes daemonset)
Collector version
v0.127.0
Environment information
Environment
OS: Linux
Compiler(if manually compiled): go 1.24.3
OpenTelemetry Collector configuration
receivers:
datadog:
endpoint: '${env:K8S_POD_IP}:8126'
read_timeout: 60s
# trace_id_cache_size: 1000 // making the cache size larger makes the crash happen after a longer amount of time, but it still happens
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 80
spike_limit_percentage: 20
batch:
send_batch_max_size: 1100
send_batch_size: 1000
timeout: 1s
exporters:
debug:
verbosity: detailed
pipelines:
traces:
exporters:
- debug
processors:
- memory_limiter
- batch
receivers:
- datadogLog output
http: panic serving xx.xx.xx.xx:<port>: runtime error: invalid memory address or nil pointer dereference
goroutine 40037 [running]:
net/http.(*conn).serve.func1()
net/http/server.go:1947 +0xbe
panic({0x4338f60?, 0x75ef000?})
runtime/panic.go:785 +0x132
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End.deferwrap1()
go.opentelemetry.io/otel/[email protected]/trace/span.go:467 +0x25
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End(0xc002efcf00, {0x0, 0x0, 0xc000dc6380?})
go.opentelemetry.io/otel/[email protected]/trace/span.go:506 +0xb7b
panic({0x4338f60?, 0x75ef000?})
runtime/panic.go:785 +0x132
github.com/hashicorp/golang-lru/v2/internal.(*LruList[...]).Remove(...)
github.com/hashicorp/golang-lru/[email protected]/internal/list.go:98
github.com/hashicorp/golang-lru/v2/simplelru.(*LRU[...]).removeElement(0x5ae94668?, 0xdd9b4b90ab87a89a?)
github.com/hashicorp/golang-lru/[email protected]/simplelru/lru.go:172 +0x3f
github.com/hashicorp/golang-lru/v2/simplelru.(*LRU[...]).removeOldest(0xc0008ff800?)
github.com/hashicorp/golang-lru/[email protected]/simplelru/lru.go:166 +0x32
github.com/hashicorp/golang-lru/v2/simplelru.(*LRU[...]).Add(0x53311a0, 0x9aa887ab904b9bdd, {0x68, 0x46, 0xe9, 0x5a, 0x0, 0x0, 0x0, 0x0, ...})
github.com/hashicorp/golang-lru/[email protected]/simplelru/lru.go:65 +0x1dd
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/datadogreceiver/internal/translator.traceID64to128(0xc0014c8a80, 0xc0005476a0)
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/[email protected]/internal/translator/traces_translator.go:97 +0x147
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/datadogreceiver/internal/translator.ToTraces(0xc00100ac00, 0xc001092c30, 0xc000c7ef00, 0xc0005476a0)
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/[email protected]/internal/translator/traces_translator.go:198 +0xf25
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/datadogreceiver.(*datadogReceiver).handleTraces(0xc000ca2b40, {0x52bcf60, 0xc00090d2c0}, 0xc000c7ef00)
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/[email protected]/receiver.go:252 +0x39b
net/http.HandlerFunc.ServeHTTP(0xc0015c89a0?, {0x52bcf60?, 0xc00090d2c0?}, 0x0?)
net/http/server.go:2220 +0x29
net/http.(*ServeMux).ServeHTTP(0xc000e77f20?, {0x52bcf60, 0xc00090d2c0}, 0xc000c7ef00)
net/http/server.go:2747 +0x1ca
go.opentelemetry.io/collector/config/confighttp.(*decompressor).ServeHTTP(0xc000e77f20, {0x52bcf60, 0xc00090d2c0}, 0xc000c7ef00)
go.opentelemetry.io/collector/config/[email protected]/compression.go:239 +0x208
go.opentelemetry.io/collector/config/confighttp.(*ServerConfig).ToServer.maxRequestBodySizeInterceptor.func2({0x52bcf60, 0xc00090d2c0}, 0xc000c7ef00)
go.opentelemetry.io/collector/config/[email protected]/confighttp.go:620 +0x87
net/http.HandlerFunc.ServeHTTP(0x52c8498?, {0x52bcf60?, 0xc00090d2c0?}, 0x5256ec0?)
net/http/server.go:2220 +0x29
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP(0xc000c3ea00, {0x52aea90, 0xc0014c89a0}, 0xc000c7edc0, {0x526ab60, 0xc0006ddbc0})
go.opentelemetry.io/contrib/instrumentation/net/http/[email protected]/handler.go:179 +0x1303
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1({0x52aea90?, 0xc0014c89a0?}, 0x1?)
go.opentelemetry.io/contrib/instrumentation/net/http/[email protected]/handler.go:67 +0x35
net/http.HandlerFunc.ServeHTTP(0xc000c7ec80?, {0x52aea90?, 0xc0014c89a0?}, 0xc00345bb20?)
net/http/server.go:2220 +0x29
go.opentelemetry.io/collector/config/confighttp.(*clientInfoHandler).ServeHTTP(0xc00099f3f8, {0x52aea90, 0xc0014c89a0}, 0xc000c7ec80)
go.opentelemetry.io/collector/config/[email protected]/clientinfohandler.go:26 +0xee
net/http.serverHandler.ServeHTTP({0x5281cf8?}, {0x52aea90?, 0xc0014c89a0?}, 0x6?)
net/http/server.go:3210 +0x8e
net/http.(*conn).serve(0xc000fa0cf0, {0x52c8498, 0xc000dfc1b0})
net/http/server.go:2092 +0x5d0
created by net/http.(*Server).Serve in goroutine 413
net/http/server.go:3360 +0x485Additional context
No response