Skip to content

Conversation

anneelisabethlelievre
Copy link
Contributor

@anneelisabethlelievre anneelisabethlelievre commented Nov 4, 2021

Signed-off-by: Anne-Elisabeth Lelièvre [email protected]

Hello, this is a fix to resolve a problem by adding exemplars to the metrics in the span metrics processor.

When I tried to send the metrics to prometheus , I got this error:

ts=2021-10-28T20:49:50.703Z caller=write_handler.go:76 level=debug component=web msg="Out of order exemplar" exemplar="{Labels:{} Value:0 Ts:0 HasTs:false}"
ts=2021-10-28T20:49:50.703Z caller=write_handler.go:121 level=warn component=web msg="Error on ingesting out-of-order exemplars" num_dropped=8

And when I tried to send to Cortex, I got this error:

{"level":"error","ts":1635446592.7831264,"caller":"exporterhelper/queued_retry.go:199","msg":"Exporting failed. The error is not retryable. Dropping data.","kind":"exporter","name":"prometheusremotewrite/0/metrics/2","error":"Permanent error: remote write returned HTTP status 400 Bad Request; err = <nil>: exemplar missing labels, timestamp: 0 series: 

By analyzing these errors, I assumed that there was a problem with the index of the exemplar data map.
So to resolve this issue, instead of adding exemplars to the a specific bucket index, I propose to simply append the exemplar data and reset the entire exemplars map so the next trace will recreate all the data structure. I have retested and everything seems to be sent correctly now on my end.

Signed-off-by: Anne-Elisabeth Lelièvre <[email protected]>

key := metricKey("metricKey")
next := new(consumertest.TracesSink)
p, err := newProcessor(zap.NewNop(), cfg, next)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest using zaptest.NewTestLogger(t) instead of the Nop logger here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Signed-off-by: Anne-Elisabeth Lelièvre <[email protected]>
Signed-off-by: Anne-Elisabeth Lelièvre <[email protected]>
@github-actions
Copy link
Contributor

This PR was marked stale due to lack of activity. It will be closed in 7 days.

@github-actions github-actions bot added the Stale label Nov 14, 2021
@anneelisabethlelievre
Copy link
Contributor Author

@kbrockhoff, @bogdandrutu This PR is ready for review if you have time to look at it, thanks.

@github-actions github-actions bot removed the Stale label Nov 16, 2021
Copy link
Contributor

@codeboten codeboten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this, pinging @albertteoh as the codeowner for this component.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 1, 2021

This PR was marked stale due to lack of activity. It will be closed in 7 days.

@github-actions github-actions bot added the Stale label Dec 1, 2021
@codeboten codeboten removed the Stale label Dec 1, 2021
@codeboten codeboten merged commit eea93ee into open-telemetry:main Dec 1, 2021
jamesmoessis pushed a commit to atlassian-forks/opentelemetry-collector-contrib that referenced this pull request Dec 8, 2021
* fix exemplars support

Signed-off-by: Anne-Elisabeth Lelièvre <[email protected]>

* switch test logger from zap.Nop to zaptest.

Signed-off-by: Anne-Elisabeth Lelièvre <[email protected]>

* fix lint error

Signed-off-by: Anne-Elisabeth Lelièvre <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants