### Component(s) exporter/prometheus ### What happened? ## Description When using the `exporter/prometheus` in OpenTelemetry Collector, exemplars are successfully generated by a manually instrumented Python application and received by the OTEL Collector. These exemplars are visible in the OTEL Collector debug logs, confirming that the trace-metric correlation works internally. However, no exemplars are being exposed via the `/metrics` endpoint scraped by Prometheus, even though all configurations are set correctly and trace IDs are present in the metric logs. ## Steps to Reproduce 1. Start the stack using the following files (all uploaded in this issue): - `docker-compose.yml` (Prometheus, Grafana) - `docker-compose.otel.yml` (OpenTelemetry Collector, Jaeger) - `otel-collector-config.yml` (OpenTelemetry Collector configuration) - `prometheus-config.yml` (Prometheus configuration) 2. Run the instrumented Python app with the command: ``` OTEL_EXEMPLARS_SAMPLING_PROBABILITY=1.0 OTEL_EXEMPLAR_FILTER=always_on python server_a.py ``` 3. While the app is running, generate traffic with the command: ``` for i in {1..6}; do curl http://localhost:6000/test; sleep 1; done ``` 4. Observe metrics in real-time: - The Python server prints metrics with exemplars in the terminal when the `/test` endpoint is hit. - Prometheus is scraping the `/metrics` endpoint from the OpenTelemetry Collector. - We are not saving the data to any files — everything is visible in live terminal output. ## Expected Result - The `/metrics` endpoint exposed by OTEL Collector should contain exemplar annotations attached to the metric lines. - Prometheus should successfully scrape and store metrics with trace exemplars for correlation. ## Actual Result - Exemplar annotations (e.g., `# {trace_id="..."}`) are missing entirely from the `/metrics` endpoint. - Prometheus receives only raw metrics with no exemplar data, making trace-metric correlation impossible. ### Collector version v0.122.1 ### Environment information ## Environment OS: macOS Sonoma 14.3.1 (Apple Silicon - M2) Docker: Docker Desktop v4.28.0 (Engine: 24.0.7, Compose: v2.24.5) Python: 3.9.18 otel-collector image: otel/opentelemetry-collector-contrib:0.122.1 Prometheus: v3.2.1 Grafana: latest (as of March 2025) Jaeger: jaegertracing/all-in-one:1.42.0 ### OpenTelemetry Collector configuration ```yaml receivers: otlp: protocols: grpc: endpoint: "0.0.0.0:4317" # Accepts traces from both local and Docker http: endpoint: "0.0.0.0:4318" # Accepts HTTP traces processors: batch: exporters: debug: verbosity: detailed otlp: endpoint: "http://jaeger:4317" tls: insecure: true prometheus: endpoint: "0.0.0.0:9464" enable_open_metrics: true service: pipelines: traces: receivers: [otlp] exporters: [debug, otlp] processors: [batch] metrics: receivers: [otlp] exporters: [debug, prometheus] processors: [batch] ``` ### Log output ```shell ### 🔹 OTEL Collector Logs (with debug enabled) ➜ prometheus-stack docker logs -f otel-collector 2025-03-22T17:50:25.473Z info service@v0.122.1/service.go:193 Setting up own telemetry... 2025-03-22T17:50:25.473Z info builders/builders.go:26 Development component. May change in the future. {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "metrics"} 2025-03-22T17:50:25.478Z info builders/builders.go:26 Development component. May change in the future. {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces"} 2025-03-22T17:50:25.480Z info service@v0.122.1/service.go:260 Starting otelcol-contrib... {"Version": "0.122.1", "NumCPU": 8} 2025-03-22T17:50:25.480Z info extensions/extensions.go:40 Starting extensions... 2025-03-22T17:50:25.481Z info otlpreceiver@v0.122.1/otlp.go:116 Starting GRPC server {"otelcol.component.id": "otlp", "otelcol.component.kind": "Receiver", "endpoint": "0.0.0.0:4317"} 2025-03-22T17:50:25.481Z info otlpreceiver@v0.122.1/otlp.go:173 Starting HTTP server {"otelcol.component.id": "otlp", "otelcol.component.kind": "Receiver", "endpoint": "0.0.0.0:4318"} 2025-03-22T17:50:25.481Z info service@v0.122.1/service.go:283 Everything is ready. Begin running and processing data. 2025-03-22T17:51:31.574Z info Metrics {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "metrics", "resource metrics": 1, "metrics": 2, "data points": 4} 2025-03-22T17:51:31.575Z info ResourceMetrics #0 Resource SchemaURL: Resource attributes: -> service.name: Str(service-a) ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope __main__ Metric #0 Descriptor: -> Name: custom_requests_total -> Description: Total custom requests -> Unit: -> DataType: Sum -> IsMonotonic: true -> AggregationTemporality: Cumulative NumberDataPoints #0 Data point attributes: -> http.method: Str(GET) -> request_id: Str(344861ac-2c09-4945-826b-37b67309bc61) StartTimestamp: 2025-03-22 17:51:30.888351 +0000 UTC Timestamp: 2025-03-22 17:51:31.55833 +0000 UTC Value: 1 Exemplars: Exemplar #0 -> Trace ID: c19a57ed96ca1db94aeb773a15145851 -> Span ID: a55aa0c81d7d9d85 -> Timestamp: 2025-03-22 17:51:30.888233 +0000 UTC -> Value: 1 NumberDataPoints #1 Data point attributes: -> http.method: Str(GET) -> request_id: Str(5efa63d8-51cd-40a8-a966-bb1400e27f65) StartTimestamp: 2025-03-22 17:51:31.440623 +0000 UTC Timestamp: 2025-03-22 17:51:31.55833 +0000 UTC Value: 1 Exemplars: Exemplar #0 -> Trace ID: 2f7f0222c12884db24145c107a537840 -> Span ID: 798c6d5f7eff0460 -> Timestamp: 2025-03-22 17:51:31.440423 +0000 UTC -> Value: 1 Metric #1 Descriptor: -> Name: custom_request_duration -> Description: Custom request duration -> Unit: ms -> DataType: Histogram -> AggregationTemporality: Cumulative HistogramDataPoints #0 Data point attributes: -> http.method: Str(GET) -> request_id: Str(344861ac-2c09-4945-826b-37b67309bc61) StartTimestamp: 2025-03-22 17:51:30.900367 +0000 UTC Timestamp: 2025-03-22 17:51:31.55833 +0000 UTC Count: 1 Sum: 12.521982 Min: 12.521982 Max: 12.521982 ExplicitBounds #0: 0.000000 ExplicitBounds #1: 5.000000 ExplicitBounds #2: 10.000000 ExplicitBounds #3: 25.000000 ExplicitBounds #4: 50.000000 ExplicitBounds #5: 75.000000 ExplicitBounds #6: 100.000000 ExplicitBounds #7: 250.000000 ExplicitBounds #8: 500.000000 ExplicitBounds #9: 750.000000 ExplicitBounds #10: 1000.000000 ExplicitBounds #11: 2500.000000 ExplicitBounds #12: 5000.000000 ExplicitBounds #13: 7500.000000 ExplicitBounds #14: 10000.000000 Buckets #0, Count: 0 Buckets #1, Count: 0 Buckets #2, Count: 0 Buckets #3, Count: 1 Buckets #4, Count: 0 Buckets #5, Count: 0 Buckets #6, Count: 0 Buckets #7, Count: 0 Buckets #8, Count: 0 Buckets #9, Count: 0 Buckets #10, Count: 0 Buckets #11, Count: 0 Buckets #12, Count: 0 Buckets #13, Count: 0 Buckets #14, Count: 0 Buckets #15, Count: 0 Exemplars: Exemplar #0 -> Trace ID: c19a57ed96ca1db94aeb773a15145851 -> Span ID: a55aa0c81d7d9d85 -> Timestamp: 2025-03-22 17:51:30.900251 +0000 UTC -> Value: 12.521982 HistogramDataPoints #1 Data point attributes: -> http.method: Str(GET) -> request_id: Str(5efa63d8-51cd-40a8-a966-bb1400e27f65) StartTimestamp: 2025-03-22 17:51:31.440736 +0000 UTC Timestamp: 2025-03-22 17:51:31.55833 +0000 UTC Count: 1 Sum: 12.529135 Min: 12.529135 Max: 12.529135 ExplicitBounds #0: 0.000000 ExplicitBounds #1: 5.000000 ExplicitBounds #2: 10.000000 ExplicitBounds #3: 25.000000 ExplicitBounds #4: 50.000000 ExplicitBounds #5: 75.000000 ExplicitBounds #6: 100.000000 ExplicitBounds #7: 250.000000 ExplicitBounds #8: 500.000000 ExplicitBounds #9: 750.000000 ExplicitBounds #10: 1000.000000 ExplicitBounds #11: 2500.000000 ExplicitBounds #12: 5000.000000 ExplicitBounds #13: 7500.000000 ExplicitBounds #14: 10000.000000 Buckets #0, Count: 0 Buckets #1, Count: 0 Buckets #2, Count: 0 Buckets #3, Count: 1 Buckets #4, Count: 0 Buckets #5, Count: 0 Buckets #6, Count: 0 Buckets #7, Count: 0 Buckets #8, Count: 0 Buckets #9, Count: 0 Buckets #10, Count: 0 Buckets #11, Count: 0 Buckets #12, Count: 0 Buckets #13, Count: 0 Buckets #14, Count: 0 Buckets #15, Count: 0 Exemplars: Exemplar #0 -> Trace ID: 2f7f0222c12884db24145c107a537840 -> Span ID: 798c6d5f7eff0460 -> Timestamp: 2025-03-22 17:51:31.440671 +0000 UTC -> Value: 12.529135 {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "metrics"} 2025-03-22T17:51:32.589Z info Metrics {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "metrics", "resource metrics": 1, "metrics": 2, "data points": 4} 2025-03-22T17:51:32.589Z info ResourceMetrics #0 Resource SchemaURL: Resource attributes: -> service.name: Str(service-a) ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope __main__ Metric #0 Descriptor: -> Name: custom_requests_total -> Description: Total custom requests -> Unit: -> DataType: Sum -> IsMonotonic: true -> AggregationTemporality: Cumulative NumberDataPoints #0 Data point attributes: -> http.method: Str(GET) -> request_id: Str(344861ac-2c09-4945-826b-37b67309bc61) StartTimestamp: 2025-03-22 17:51:30.888351 +0000 UTC Timestamp: 2025-03-22 17:51:32.571423 +0000 UTC Value: 1 NumberDataPoints #1 Data point attributes: -> http.method: Str(GET) -> request_id: Str(5efa63d8-51cd-40a8-a966-bb1400e27f65) StartTimestamp: 2025-03-22 17:51:31.440623 +0000 UTC Timestamp: 2025-03-22 17:51:32.571423 +0000 UTC Value: 1 Metric #1 Descriptor: -> Name: custom_request_duration -> Description: Custom request duration -> Unit: ms -> DataType: Histogram -> AggregationTemporality: Cumulative HistogramDataPoints #0 Data point attributes: -> http.method: Str(GET) -> request_id: Str(344861ac-2c09-4945-826b-37b67309bc61) StartTimestamp: 2025-03-22 17:51:30.900367 +0000 UTC Timestamp: 2025-03-22 17:51:32.571423 +0000 UTC Count: 1 Sum: 12.521982 Min: 12.521982 Max: 12.521982 ExplicitBounds #0: 0.000000 ExplicitBounds #1: 5.000000 ExplicitBounds #2: 10.000000 ExplicitBounds #3: 25.000000 ExplicitBounds #4: 50.000000 ExplicitBounds #5: 75.000000 ExplicitBounds #6: 100.000000 ExplicitBounds #7: 250.000000 ExplicitBounds #8: 500.000000 ExplicitBounds #9: 750.000000 ExplicitBounds #10: 1000.000000 ExplicitBounds #11: 2500.000000 ExplicitBounds #12: 5000.000000 ExplicitBounds #13: 7500.000000 ExplicitBounds #14: 10000.000000 Buckets #0, Count: 0 Buckets #1, Count: 0 Buckets #2, Count: 0 Buckets #3, Count: 1 Buckets #4, Count: 0 Buckets #5, Count: 0 Buckets #6, Count: 0 Buckets #7, Count: 0 Buckets #8, Count: 0 Buckets #9, Count: 0 Buckets #10, Count: 0 Buckets #11, Count: 0 Buckets #12, Count: 0 Buckets #13, Count: 0 Buckets #14, Count: 0 Buckets #15, Count: 0 HistogramDataPoints #1 Data point attributes: -> http.method: Str(GET) -> request_id: Str(5efa63d8-51cd-40a8-a966-bb1400e27f65) StartTimestamp: 2025-03-22 17:51:31.440736 +0000 UTC Timestamp: 2025-03-22 17:51:32.571423 +0000 UTC Count: 1 Sum: 12.529135 Min: 12.529135 Max: 12.529135 ExplicitBounds #0: 0.000000 ExplicitBounds #1: 5.000000 ExplicitBounds #2: 10.000000 ExplicitBounds #3: 25.000000 ExplicitBounds #4: 50.000000 ExplicitBounds #5: 75.000000 ExplicitBounds #6: 100.000000 ExplicitBounds #7: 250.000000 ExplicitBounds #8: 500.000000 ExplicitBounds #9: 750.000000 ExplicitBounds #10: 1000.000000 ExplicitBounds #11: 2500.000000 ExplicitBounds #12: 5000.000000 ExplicitBounds #13: 7500.000000 ExplicitBounds #14: 10000.000000 Buckets #0, Count: 0 Buckets #1, Count: 0 Buckets #2, Count: 0 Buckets #3, Count: 1 Buckets #4, Count: 0 Buckets #5, Count: 0 Buckets #6, Count: 0 Buckets #7, Count: 0 Buckets #8, Count: 0 Buckets #9, Count: 0 Buckets #10, Count: 0 Buckets #11, Count: 0 Buckets #12, Count: 0 Buckets #13, Count: 0 Buckets #14, Count: 0 Buckets #15, Count: 0 {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "metrics"} 2025-03-22T17:51:33.601Z info Metrics {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "metrics", "resource metrics": 1, "metrics": 2, "data points": 4} 2025-03-22T17:51:33.601Z info Traces {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces", "resource spans": 1, "spans": 2} 2025-03-22T17:51:33.602Z info ResourceMetrics #0 Resource SchemaURL: Resource attributes: -> service.name: Str(service-a) ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope __main__ Metric #0 Descriptor: -> Name: custom_requests_total -> Description: Total custom requests -> Unit: -> DataType: Sum -> IsMonotonic: true -> AggregationTemporality: Cumulative NumberDataPoints #0 Data point attributes: -> http.method: Str(GET) -> request_id: Str(344861ac-2c09-4945-826b-37b67309bc61) StartTimestamp: 2025-03-22 17:51:30.888351 +0000 UTC Timestamp: 2025-03-22 17:51:33.580137 +0000 UTC Value: 1 NumberDataPoints #1 Data point attributes: -> http.method: Str(GET) -> request_id: Str(5efa63d8-51cd-40a8-a966-bb1400e27f65) StartTimestamp: 2025-03-22 17:51:31.440623 +0000 UTC Timestamp: 2025-03-22 17:51:33.580137 +0000 UTC Value: 1 Metric #1 Descriptor: -> Name: custom_request_duration -> Description: Custom request duration -> Unit: ms -> DataType: Histogram -> AggregationTemporality: Cumulative HistogramDataPoints #0 Data point attributes: -> http.method: Str(GET) -> request_id: Str(344861ac-2c09-4945-826b-37b67309bc61) StartTimestamp: 2025-03-22 17:51:30.900367 +0000 UTC Timestamp: 2025-03-22 17:51:33.580137 +0000 UTC Count: 1 Sum: 12.521982 Min: 12.521982 Max: 12.521982 ExplicitBounds #0: 0.000000 ExplicitBounds #1: 5.000000 ExplicitBounds #2: 10.000000 ExplicitBounds #3: 25.000000 ExplicitBounds #4: 50.000000 ExplicitBounds #5: 75.000000 ExplicitBounds #6: 100.000000 ExplicitBounds #7: 250.000000 ExplicitBounds #8: 500.000000 ExplicitBounds #9: 750.000000 ExplicitBounds #10: 1000.000000 ExplicitBounds #11: 2500.000000 ExplicitBounds #12: 5000.000000 ExplicitBounds #13: 7500.000000 ExplicitBounds #14: 10000.000000 Buckets #0, Count: 0 Buckets #1, Count: 0 Buckets #2, Count: 0 Buckets #3, Count: 1 Buckets #4, Count: 0 Buckets #5, Count: 0 Buckets #6, Count: 0 Buckets #7, Count: 0 Buckets #8, Count: 0 Buckets #9, Count: 0 Buckets #10, Count: 0 Buckets #11, Count: 0 Buckets #12, Count: 0 Buckets #13, Count: 0 Buckets #14, Count: 0 Buckets #15, Count: 0 HistogramDataPoints #1 Data point attributes: -> http.method: Str(GET) -> request_id: Str(5efa63d8-51cd-40a8-a966-bb1400e27f65) StartTimestamp: 2025-03-22 17:51:31.440736 +0000 UTC Timestamp: 2025-03-22 17:51:33.580137 +0000 UTC Count: 1 Sum: 12.529135 Min: 12.529135 Max: 12.529135 ExplicitBounds #0: 0.000000 ExplicitBounds #1: 5.000000 ExplicitBounds #2: 10.000000 ExplicitBounds #3: 25.000000 ExplicitBounds #4: 50.000000 ExplicitBounds #5: 75.000000 ExplicitBounds #6: 100.000000 ExplicitBounds #7: 250.000000 ExplicitBounds #8: 500.000000 ExplicitBounds #9: 750.000000 ExplicitBounds #10: 1000.000000 ExplicitBounds #11: 2500.000000 ExplicitBounds #12: 5000.000000 ExplicitBounds #13: 7500.000000 ExplicitBounds #14: 10000.000000 Buckets #0, Count: 0 Buckets #1, Count: 0 Buckets #2, Count: 0 Buckets #3, Count: 1 Buckets #4, Count: 0 Buckets #5, Count: 0 Buckets #6, Count: 0 Buckets #7, Count: 0 Buckets #8, Count: 0 Buckets #9, Count: 0 Buckets #10, Count: 0 Buckets #11, Count: 0 Buckets #12, Count: 0 Buckets #13, Count: 0 Buckets #14, Count: 0 Buckets #15, Count: 0 {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "metrics"} 2025-03-22T17:51:33.602Z info ResourceSpans #0 Resource SchemaURL: Resource attributes: -> service.name: Str(service-a) ScopeSpans #0 ScopeSpans SchemaURL: InstrumentationScope __main__ Span #0 Trace ID : c19a57ed96ca1db94aeb773a15145851 Parent ID : ID : a55aa0c81d7d9d85 Name : custom-operation Kind : Internal Start time : 2025-03-22 17:51:30.875631 +0000 UTC End time : 2025-03-22 17:51:30.900463 +0000 UTC Status code : Unset Status message : Attributes: -> http.method: Str(GET) Span #1 Trace ID : 2f7f0222c12884db24145c107a537840 Parent ID : ID : 798c6d5f7eff0460 Name : custom-operation Kind : Internal Start time : 2025-03-22 17:51:31.4278 +0000 UTC End time : 2025-03-22 17:51:31.440866 +0000 UTC Status code : Unset Status message : Attributes: -> http.method: Str(GET) {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces"} 2025-03-22T17:51:34.011Z info Metrics {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "metrics", "resource metrics": 1, "metrics": 2, "data points": 4} 2025-03-22T17:51:34.011Z info ResourceMetrics #0 Resource SchemaURL: Resource attributes: -> service.name: Str(service-a) ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope __main__ Metric #0 Descriptor: -> Name: custom_requests_total -> Description: Total custom requests -> Unit: -> DataType: Sum -> IsMonotonic: true -> AggregationTemporality: Cumulative NumberDataPoints #0 Data point attributes: -> http.method: Str(GET) -> request_id: Str(344861ac-2c09-4945-826b-37b67309bc61) StartTimestamp: 2025-03-22 17:51:30.888351 +0000 UTC Timestamp: 2025-03-22 17:51:33.855355 +0000 UTC Value: 1 NumberDataPoints #1 Data point attributes: -> http.method: Str(GET) -> request_id: Str(5efa63d8-51cd-40a8-a966-bb1400e27f65) StartTimestamp: 2025-03-22 17:51:31.440623 +0000 UTC Timestamp: 2025-03-22 17:51:33.855355 +0000 UTC Value: 1 Metric #1 Descriptor: -> Name: custom_request_duration -> Description: Custom request duration -> Unit: ms -> DataType: Histogram -> AggregationTemporality: Cumulative HistogramDataPoints #0 Data point attributes: -> http.method: Str(GET) -> request_id: Str(344861ac-2c09-4945-826b-37b67309bc61) StartTimestamp: 2025-03-22 17:51:30.900367 +0000 UTC Timestamp: 2025-03-22 17:51:33.855355 +0000 UTC Count: 1 Sum: 12.521982 Min: 12.521982 Max: 12.521982 ExplicitBounds #0: 0.000000 ExplicitBounds #1: 5.000000 ExplicitBounds #2: 10.000000 ExplicitBounds #3: 25.000000 ExplicitBounds #4: 50.000000 ExplicitBounds #5: 75.000000 ExplicitBounds #6: 100.000000 ExplicitBounds #7: 250.000000 ExplicitBounds #8: 500.000000 ExplicitBounds #9: 750.000000 ExplicitBounds #10: 1000.000000 ExplicitBounds #11: 2500.000000 ExplicitBounds #12: 5000.000000 ExplicitBounds #13: 7500.000000 ExplicitBounds #14: 10000.000000 Buckets #0, Count: 0 Buckets #1, Count: 0 Buckets #2, Count: 0 Buckets #3, Count: 1 Buckets #4, Count: 0 Buckets #5, Count: 0 Buckets #6, Count: 0 Buckets #7, Count: 0 Buckets #8, Count: 0 Buckets #9, Count: 0 Buckets #10, Count: 0 Buckets #11, Count: 0 Buckets #12, Count: 0 Buckets #13, Count: 0 Buckets #14, Count: 0 Buckets #15, Count: 0 HistogramDataPoints #1 Data point attributes: -> http.method: Str(GET) -> request_id: Str(5efa63d8-51cd-40a8-a966-bb1400e27f65) StartTimestamp: 2025-03-22 17:51:31.440736 +0000 UTC Timestamp: 2025-03-22 17:51:33.855355 +0000 UTC Count: 1 Sum: 12.529135 Min: 12.529135 Max: 12.529135 ExplicitBounds #0: 0.000000 ExplicitBounds #1: 5.000000 ExplicitBounds #2: 10.000000 ExplicitBounds #3: 25.000000 ExplicitBounds #4: 50.000000 ExplicitBounds #5: 75.000000 ExplicitBounds #6: 100.000000 ExplicitBounds #7: 250.000000 ExplicitBounds #8: 500.000000 ExplicitBounds #9: 750.000000 ExplicitBounds #10: 1000.000000 ExplicitBounds #11: 2500.000000 ExplicitBounds #12: 5000.000000 ExplicitBounds #13: 7500.000000 ExplicitBounds #14: 10000.000000 Buckets #0, Count: 0 Buckets #1, Count: 0 Buckets #2, Count: 0 Buckets #3, Count: 1 Buckets #4, Count: 0 Buckets #5, Count: 0 Buckets #6, Count: 0 Buckets #7, Count: 0 Buckets #8, Count: 0 Buckets #9, Count: 0 Buckets #10, Count: 0 Buckets #11, Count: 0 Buckets #12, Count: 0 Buckets #13, Count: 0 Buckets #14, Count: 0 Buckets #15, Count: 0 {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "metrics"} --- ### 🔹 Prometheus /metrics API Output (filtered) ➜ ~ curl http://localhost:9464/metrics # HELP custom_request_duration_milliseconds Custom request duration # TYPE custom_request_duration_milliseconds histogram custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="0"} 0 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="5"} 0 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="10"} 0 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="25"} 1 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="50"} 1 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="75"} 1 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="100"} 1 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="250"} 1 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="500"} 1 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="750"} 1 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="1000"} 1 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="2500"} 1 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="5000"} 1 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="7500"} 1 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="10000"} 1 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61",le="+Inf"} 1 custom_request_duration_milliseconds_sum{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61"} 12.521982192993164 custom_request_duration_milliseconds_count{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61"} 1 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="0"} 0 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="5"} 0 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="10"} 0 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="25"} 1 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="50"} 1 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="75"} 1 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="100"} 1 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="250"} 1 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="500"} 1 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="750"} 1 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="1000"} 1 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="2500"} 1 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="5000"} 1 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="7500"} 1 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="10000"} 1 custom_request_duration_milliseconds_bucket{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65",le="+Inf"} 1 custom_request_duration_milliseconds_sum{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65"} 12.529134750366211 custom_request_duration_milliseconds_count{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65"} 1 # HELP custom_requests_total Total custom requests # TYPE custom_requests_total counter custom_requests_total{http_method="GET",job="service-a",request_id="344861ac-2c09-4945-826b-37b67309bc61"} 1 custom_requests_total{http_method="GET",job="service-a",request_id="5efa63d8-51cd-40a8-a966-bb1400e27f65"} 1 ``` ### Additional context ## Additional Context - Tried multiple versions of Prometheus (`v2.50.0`, `v3.2.1`) and OpenTelemetry Collector (`v0.122.1`). - Verified that the exemplar-related OTEL environment variables were correctly set: - `OTEL_EXEMPLARS_SAMPLING_PROBABILITY=1.0` - `OTEL_EXEMPLAR_FILTER=always_on` - Metrics with exemplars are confirmed to be emitted in OTEL collector logs while the request is in flight. - Prometheus `/metrics` scrape does not contain any exemplar lines even during active request traffic. - Prometheus is running with `--enable-feature=exemplar-storage` and `--enable-feature=otlp-write-receiver`. - Exemplar fields are printed in OTEL logs, but they don’t make it to Prometheus scrape endpoint. - Correlation attempt was done using **manual instrumentation** with OpenTelemetry SDK (Python). - Used two custom metrics: - Counter: `custom_requests_total` - Histogram: `custom_request_duration` - Both metrics include the following attributes: - `http.method` (example: `"GET"`) - `request_id` (example: `uuid.uuid4()` to avoid exemplar overwriting) - Ensured that: - Prometheus scrape interval is set to `1s` - OTEL export interval is set to `1s` - Auto-instrumentation failed to emit exemplars previously; switched to manual instrumentation for clarity. - Attempted to use `enable_open_metrics: true` and `enable_exemplars: true` in Prometheus exporter config. However, the collector failed to start when `enable_exemplars: true` was added. ### 🐍 Python Server Instrumentation Summary The Flask app is manually instrumented for both traces and metrics using the OpenTelemetry SDK. Custom metrics and exemplars are emitted using the /test endpoint. A snippet of the relevant /test route logic that emits exemplars: - `Histogram`: custom_request_duration - `Counter`: custom_requests_total - `request_id` attribute is added to both metrics and spans Tracer and Meter setup with: - `OTLPSpanExporter` (to Jaeger) - `OTLPMetricExporter` (to Collector) - `ConsoleMetricExporter` for debug output 📦 **Attached Archive:** `exemplar-correlation-setup.zip` This archive includes all relevant files used to reproduce the issue: - `server_a.py`: Manually instrumented Python app with traces + metrics - `otel-collector-config.yml`: OTEL config with Prometheus + Jaeger exporters - `docker-compose.yml`: Prometheus, Grafana, Node Exporter - `docker-compose.otel.yml`: OTEL Collector and Jaeger - `prometheus.yml`: Scrape configuration with 1s interval [exemplar-correlation-setup.zip](https://github.com/user-attachments/files/19404909/exemplar-correlation-setup.zip)