-
Notifications
You must be signed in to change notification settings - Fork 1k
Description
Describe the bug
After upgrading the Datadog Helm chart from 3.117.3 to 3.118.3, we observed that APM traces from ingress-nginx (version 4.11.5, which uses OpenTelemetry) are no longer being reported correctly to Datadog APM. Specifically:
With chart 3.117.3: traces were received using the nginx.server span name via OpenTelemetry.
With chart 3.118.3: those traces are no longer received. Instead, we are only getting old-style http.server.request metrics (as seen with older ingress-nginx versions like 4.9.0).
We suspect a new version breaking change in the chart version 3.118.3 that affects OTLP receiver or span name translation logic.
To Reproduce
Steps to reproduce the behavior:
- Deploy ingress-nginx version 4.11.5 with OpenTelemetry enabled and properly configured OTEL_RESOURCE_ATTRIBUTES.
- Deploy Datadog Helm chart 3.117.3 with datadog.apm.enabled=true and OTLP receiver enabled (gRPC port 4317).
- Verify APM traces with span name nginx.server are visible in Datadog APM.
- Upgrade to chart version 3.118.3.
- Observe that the nginx.server traces disappear, and only http.server.request style metrics are visible.
Expected behavior
OpenTelemetry-based APM traces from ingress-nginx (v4.11.5) using the nginx.server span name should continue to be received and displayed in Datadog APM after chart upgrade.
Versions
- Ingress-nginx: 4.11.5
- Datadog Helm chart: 3.118.3 (regression), 3.117.3 (working)
- Datadog Agent image: default for Helm chart versions
- Kubernetes: v1.28