Add section on how to build an otelcol dist #1234

jpkrohling · 2020-10-09T14:04:19Z

Signed-off-by: Juraci Paixão Kröhling [email protected]

Description: Added readme section with links to resources on how to create custom OpenTelemetry Collector distributions.

Link to tracking Issue: open-telemetry/opentelemetry-collector#213

Testing: n/a

Documentation: n/a

README.md

Signed-off-by: Juraci Paixão Kröhling <[email protected]>

owais · 2020-10-12T16:42:18Z

README.md

+## Creating your own OpenTelemetry Collector distribution
+
+A guide on how to create your own distribution is available in this blog post: 
+["Building your own OpenTelemetry Collector distribution"](https://medium.com/p/42337e994b63)


I may be wrong but isn't medium content sometimes locked behind a paywall now? Not sure if a freely available project both as in beer and speech should rely on extended documentation that might not be accessible to everyone.

Nothing against medium or content they host. Just not sure if OpenTelemetry as a product should direct users to content/documentation/guides that they might not be able to read.

Great article BTW @jpkrohling

Oh, I see this is the official OpenTelemetry blog on medium. I suppose this decision has already been taken then by the maintainers and we are fine with publishin there? Please disregard if that's the case.

sometimes locked behind a paywall now

Each author/publication can choose to make money on Medium and have a paywall in front of individual posts, or to allow all users to have free access to the content, without any paywalls. I always publish articles without opting-in for the paywall, and I believe the OpenTelemetry publication also doesn't.

README.md

Updates #1234 Signed-off-by: Bogdan Drutu <[email protected]>

… add 'messaging.url' span attribute (#1234) * boto3sqs: Fix various issues * do not use 'otel' prefix for propagation keys to make propagation compatible with other SQS instrumentations like Node.Js Inject propergator.fields keys into the MessageAttributeNames argument for 'receive_message' calls to retreive the corresponding message attributes * add 'messaging.url' span attribute to SQS spans * add boto3sqs instrumentation to tox.ini to run tests in CI * add some basic unit tests * changelog * fix linting issues * unset instrumented flag on uninstrument

#### Description Removes `cmd/oteltestbedcol` from list since it has no Makefile  #### Link to tracking issue Fixes #35606

)  #### Description Removes `cmd/oteltestbedcol` from list since it has no Makefile  #### Link to tracking issue Fixes open-telemetry#35606

…atus (#35648)  #### Description Refactors the `extension/healthcheckv2extension/internal/status` into `pkg/status`. This exposes the aggregator to be used by other extensions to gather component status information using the `extension.StatusWatcher`.  #### Link to tracking issue Closes #34692  #### Testing Being it was a refactor and all the same tests provided coverage, no additional testing was added.  #### Documentation Added a `README.md` to the `pkg/status` to provide information on where this package can be used. --------- Co-authored-by: Matthew Wear <[email protected]>

#### Description Update OTel mode to implementation to serialize log body into body.* fields  #### Link to tracking issue  #### Testing  #### Documentation

…rning error to avoid upstream retries (#35740)  #### Description Log metrics validation error instead of returning to avoid upstream retries  #### Link to tracking issue  #### Testing  #### Documentation

#### Description  #### Link to tracking issue #35741 Fixes Wrong comparison func implement in translator/prometheusremotewrite.  #### Testing  #### Documentation  Signed-off-by: Juraj Michalek <[email protected]>

#### Description Follow up from #35703. Draft starting the work on adding support for remote write 2.0 in the translation package. Adding support for translating gauges. This is first iteration and to keep the PR small * we don't handle duplicate metrics * only support gauges * don't handle other labels than metric name * don't handle exemplars * don't handle metadata  #### Link to tracking issue #33661 Fixes  #### Testing  #### Documentation  --------- Signed-off-by: Juraj Michalek <[email protected]> Co-authored-by: Arthur Silva Sens <[email protected]> Co-authored-by: David Ashpole <[email protected]>

#### Description Follow [the contract](https://github.com/open-telemetry/opentelemetry-collector/blob/df3c9e38a80ccc3b14705462be2e2e51c628a3b3/receiver/doc.go#L10) for cloudflare receiver  #### Link to tracking issue Related #5909  #### Testing Added

#### Description Add a new receiver to handle systemd service state reporting as metrics.  #### Link to tracking issue #33532

… prefix conflict in OTel mapping mode (#35651)  #### Description Metric names should be flattened and exported as is, even when one metric name is a prefix of another. Same for attributes for all logs, metrics and traces.  #### Link to tracking issue  #### Testing  #### Documentation

#### Description Version of MySQL older than 8.0.22 use the `show slave status` vs `show replica status` view to retrieve replication metrics. This allows for both the newer & older views to work and retrieve data for the two actual replica metrics available (`mysql.replica.time_behind_source` & `mysql.replica.sql_delay`).  #### Link to tracking issue Fixes #35217  #### Testing Manual testing against MySQL versions 8.0, 8.0.37, & MariaDB Ver 15.1 Distrib 10.5.26-MariaDB  #### Documentation None needed

#### Description Adds initial implementation of ntpreceiver.  #### Link to tracking issue #34375

…only (#35839)  #### Description Make OTel mapping mode use RequireDataStream in docappender, which means it will only send to data streams. This prevents auto creating regular indices in OTel mapping mode due to a race condition in Elasticsearch where otel-data index templates are not ready.  #### Link to tracking issue  #### Testing  #### Documentation

#### Description Enable gzip compression by default, at hardcoded level BestSpeed. To disable compression, set `compression` to `none`.  #### Link to tracking issue  #### Testing  #### Documentation

#### Description Fixes an issue where the Kafka receiver would block on shutdown. There was an earlier fix for this issue [here](#32720). This does solve the issue, but it was only applied to the traces receiver, not the logs or metrics receiver. The issue is this go routine in the `Start()` functions for logs and metrics: ```go go func() { if err := c.consumeLoop(ctx, metricsConsumerGroup); err != nil { componentstatus.ReportStatus(host, componentstatus.NewFatalErrorEvent(err)) } }() ``` The `consumeLoop()` function returns a `context.Canceled` error when `Shutdown()` is called, which is expected. However `componentstatus.ReportStatus()` blocks while attempting to report this error. The reason/bug for this can be found [here](open-telemetry/opentelemetry-collector#9824). The previously mentioned PR fixed this for the traces receiver by checking if the error returned by `consumeLoop()` is `context.Canceled`: ```go go func() { if err := c.consumeLoop(ctx, consumerGroup); !errors.Is(err, context.Canceled) { componentstatus.ReportStatus(host, componentstatus.NewFatalErrorEvent(err)) } }() ``` Additionally, this is `consumeLoop()` for the traces receiver, with the logs and metrics versions being identical: ```go func (c *kafkaTracesConsumer) consumeLoop(ctx context.Context, handler sarama.ConsumerGroupHandler) error { for { // `Consume` should be called inside an infinite loop, when a // server-side rebalance happens, the consumer session will need to be // recreated to get the new claims if err := c.consumerGroup.Consume(ctx, c.topics, handler); err != nil { c.settings.Logger.Error("Error from consumer", zap.Error(err)) } // check if context was cancelled, signaling that the consumer should stop if ctx.Err() != nil { c.settings.Logger.Info("Consumer stopped", zap.Error(ctx.Err())) return ctx.Err() } } } ``` This does fix the issue, however the only error that can be returned by `consumeLoop()` is a canceled context. When we create the context and cancel function, we use `context.Background()`: ```go ctx, cancel := context.WithCancel(context.Background()) ``` This context is only used by `consumeLoop()` and the cancel function is only called in `Shutdown()`. Because `consumeLoop()` can only return a `context.Canceled` error, this PR removes this unused code for the logs, metrics, and traces receivers. Instead, `consumeLoop()` still logs the `context.Canceled` error but it does not return any error and the go routine simply just calls `consumeLoop()`. Additional motivation for removing the call to `componentstatus.ReportStatus()` is the underlying function called by it, `componentstatus.Report()` says it does not need to be called during `Shutdown()` or `Start()` as the service already does so for the given component, [comment here](https://github.com/open-telemetry/opentelemetry-collector/blob/main/component/componentstatus/status.go#L21-L25). Even if there wasn't a bug causing this call to block, the component still shouldn't call it since it would only be called during `Shutdown()`.  #### Link to tracking issue Fixes #30789  #### Testing Tested in a build of the collector with these changes scraping logs from a Kafka instance. When the collector is stopped and `Shutdown()` gets called, the receiver did not block and the collector stopped gracefully as expected.

…rprisereceiver (open-telemetry#42631)  #### Description This PR adds a new metric to splunkenterprisereceiver called `splunk.license.expiration.seconds_remaining` with three attributes, `splunk.license.status`, `splunk.license.label`, and `splunk.license.type`.  #### Link to tracking issue Fixes open-telemetry#42630  #### Testing Tested on a Splunk stack, and retrieved metrics in Splunk O11y.  #### Documentation Added the standard documentation that outlines the metric and its attributes.

#### Description `TestLRUSetLifeTime` is a flaky test, and the root cause seems to be that the expire feature uses the wall clock to remove items. Prior attempts to fix this test haven't been successful. Taking a step back and looking into the test, it seems we are testing the `lifetime` feature of the go-freelru library more than our own usage of the library. Creating this PR to drop the test, as I feel it is out of scope of the exporter. Happy to discuss if others feel this is a required test.  #### Link to tracking issue Fixes open-telemetry#43153  #### Testing N/A  #### Documentation N/A

…-telemetry#43117) #### Description Fixed one of the errors reported in open-telemetry#43044 Two goleak errors I faced from the `recombine` package. One of the errors indicates that `flushLoop` is still running on a goroutine after test finished. This is due to the benchmark tests never call operator's `Stop()`(More precisely, `Transformer.chClose` is never closed), so `flushLoop()` indefinitely runs. Calling `op.Stop()` makes it silent. ``` goleak: Errors on successful test run: found unexpected goroutines: [Goroutine 61 in state select, with github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/recombine.(*Transformer).flushLoop on top of the stack: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/recombine.(*Transformer).flushLoop(0x14000188a80) /Users/ray.kang/github.com/gnak-yar/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/recombine/transformer.go:62 +0x64 created by github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/recombine.(*Transformer).Start in goroutine 104 /Users/ray.kang/github.com/gnak-yar/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/recombine/transformer.go:56 +0x5c ``` The other error occurs because there's an infinite loop receiving from a channel of `FakeOutput`. Closing the channel resolves the error. ``` Goroutine 103 in state chan receive, with github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/recombine.BenchmarkRecombine.func1 on top of the stack: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/recombine.BenchmarkRecombine.func1() /Users/ray.kang/github.com/gnak-yar/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/recombine/transformer_test.go:795 +0x2c created by github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/recombine.BenchmarkRecombine in goroutine 21 /Users/ray.kang/github.com/gnak-yar/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/recombine/transformer_test.go:793 +0x498 ```  #### Link to tracking issue Fixes open-telemetry#43044  #### Testing Unit tests

…from metrics (open-telemetry#43150)  #### Description `signaltometrics.service.*` fields were added to the metrics produced from signaltometrics component to ensure single-writer is obeyed ([ref](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/signaltometricsconnector#single-writer)). However, we don't require all the fields to ensure this. Based on the description of `service.instance.id` it is alone enough to ensure single-writer ([ref](https://opentelemetry.io/docs/specs/semconv/registry/attributes/service/)). The other fields leak information about the service which is not desirable and also unnecessary for our purpose.  #### Link to tracking issue Fixes open-telemetry#43148  #### Testing Updated in PR  #### Documentation Updated  --------- Co-authored-by: Gil Raphaelli <[email protected]>

…ration (open-telemetry#42924)  #### Description The container parser's `add_metadata_from_filepath` feature requires `include_file_path` to be enabled([docs](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/docs/operators/container.md#add-metadata-from-file-path)). When `include_file_path` is disabled, users encounter an error message, `type '<nil>' cannot be parsed as log path field`, which doesn't clearly indicate the misconfiguration. This change provides users a better explanation of the error. Ideally, this misconfiguration should be detected during initialization to fail fast. However, it may require significant changes since operator configurations (file input, container parser, etc.) are separated and highly abstracted. It seems like this approach is the simplest solution for now.  #### Link to tracking issue open-telemetry#42859  #### Testing Added a unit test for the case. --------- Co-authored-by: Christos Markou <[email protected]>

#### Description Currently, the redisreceiver fetches only from the INFO command (and is capable of monitoring individual redis performance). We need to record from the CLUSTER INFO command too.  #### Link to tracking issue Fixes open-telemetry#38117  #### Testing Added tests for the `CLUSTER INFO` command in the `receiver/redisreceiver/client_test.go` file. However, the `cluster_info.txt` is yet to be added.  --------- Signed-off-by: Yaten <[email protected]>

…emetry#42767)  #### Description Removes the deprecated `batcher` configuration and makes `sending_queue` as default. To adhere to the previous behaviour, the sending queue defaults are kept as close to the current defaults as possible. The PR also drops `asyncBulkIndexer` and related code, a future PR will follow to refactor the code around bulk indexer sessions.  #### Link to tracking issue Part of open-telemetry#42718  #### Testing Updated  #### Documentation Updated  --------- Co-authored-by: Carson Ip <[email protected]>

…pen-telemetry#42805)  #### Description Some dependencies of datadogexporter are other components, which themselves are bringing over dependencies to anyone using this exporter. This PR moves the e2e tests to a separate module to avoid this complexity.  #### Link to tracking issue Fixes open-telemetry#42591

…etry#42955)  #### Description open-telemetry/opentelemetry-collector#13756 added support for exposing metrics' stability level in the generated documentation. This PR makes use of this functionality. We start by setting the stability of all metrics to `development`. More info about levels can be found at https://github.com/open-telemetry/opentelemetry-specification/blob/v1.49.0/oteps/0232-maturity-of-otel.md#maturity-levels. Related to: - open-telemetry/opentelemetry-collector#11878 - open-telemetry/opentelemetry-collector#13297 - open-telemetry#35325 - open-telemetry#42809  #### Link to tracking issue Fixes ~  #### Testing ~  #### Documentation Updated  Signed-off-by: ChrsMark <[email protected]>

#### Description  Updates build-tools to v0.28.1 (except for checkapi). See https://github.com/open-telemetry/opentelemetry-go-build-tools/releases/tag/v0.28.1 for details.

…om Log_Content, LogTag, Log,LogGroupList, LogGroup (open-telemetry#43212)  #### Description Remove public visibility from Log_Content, LogTag, Log,LogGroupList, LogGroup  #### Link to tracking issue Fixes open-telemetry#43138  #### Testing  #### Documentation

…nnel (open-telemetry#43221)  #### Description Adds reference to this new channel for new contributors  #### Link to tracking issue Updates open-telemetry/sig-contributor-experience/issues/55

…messages to use microsecond precision (open-telemetry#43211)  #### Description This PR fixes the timestamp format used for `rfc5424` format to be with the maximum of microsecond precision instead of nanosecond as defined by the [RFC5424 spec](https://www.rfc-editor.org/rfc/rfc5424#section-6).  #### Link to tracking issue Fixes open-telemetry#43114  #### Testing Updated tests and added a new unit test to ensure the timestamp is always exported with the maximum of microsecond precision. --------- Signed-off-by: Paulo Dias <[email protected]> Co-authored-by: Andrzej Stencel <[email protected]>

…pen-telemetry#43213)  #### Description Another fix for open-telemetry#43044. This error is not reported in the issue. ``` goos: darwin goarch: arm64 pkg: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/regex cpu: Apple M4 Max BenchmarkParseNoCache-14 15130 78689 ns/op BenchmarkParseWithMemoryCache-14 59836 19746 ns/op BenchmarkParseWithMemoryCacheFullByOne-14 53994 20786 ns/op BenchmarkParseWithMemoryCacheFullBy10-14 38668 30696 ns/op BenchmarkParseWithMemoryCacheFullBy50-14 20751 57883 ns/op BenchmarkParseWithMemoryCacheFullBy90-14 15634 75974 ns/op BenchmarkParseWithMemoryCacheFullBy99-14 14929 79251 ns/op BenchmarkParseNoCacheOneFile-14 663885 1735 ns/op BenchmarkParseWithMemoryCacheOneFile-14 91589362 13.04 ns/op PASS goleak: Errors on successful test run: found unexpected goroutines: [Goroutine 26460181 in state select, with github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/regex.(*atomicLimiter).init.func1.1 on top of the stack: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/regex.(*atomicLimiter).init.func1.1() /Users/ray.kang/github.com/gnak-yar/opentelemetry-collector-contrib/pkg/stanza/operator/parser/regex/cache.go:171 +0x94 created by github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/regex.(*atomicLimiter).init.func1 in goroutine 26461891 /Users/ray.kang/github.com/gnak-yar/opentelemetry-collector-contrib/pkg/stanza/operator/parser/regex/cache.go:165 +0x60 Goroutine 10525566 in state select, with github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/regex.(*atomicLimiter).init.func1.1 on top of the stack: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/regex.(*atomicLimiter).init.func1.1() /Users/ray.kang/github.com/gnak-yar/opentelemetry-collector-contrib/pkg/stanza/operator/parser/regex/cache.go:171 +0x94 created by github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/regex.(*atomicLimiter).init.func1 in goroutine 9517088 /Users/ray.kang/github.com/gnak-yar/opentelemetry-collector-contrib/pkg/stanza/operator/parser/regex/cache.go:165 +0x60 Goroutine 9526274 in state select, with github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/regex.(*atomicLimiter).init.func1.1 on top of the stack: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/regex.(*atomicLimiter).init.func1.1() /Users/ray.kang/github.com/gnak-yar/opentelemetry-collector-contrib/pkg/stanza/operator/parser/regex/cache.go:171 +0x94 created by github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/regex.(*atomicLimiter).init.func1 in goroutine 9517088 /Users/ray.kang/github.com/gnak-yar/opentelemetry-collector-contrib/pkg/stanza/operator/parser/regex/cache.go:165 +0x60 Goroutine 26472023 in state select, with github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/regex.(*atomicLimiter).init.func1.1 on top of the stack: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/regex.(*atomicLimiter).init.func1.1() /Users/ray.kang/github.com/gnak-yar/opentelemetry-collector-contrib/pkg/stanza/operator/parser/regex/cache.go:171 +0x94 created by github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/regex.(*atomicLimiter).init.func1 in goroutine 26461891 ... ``` Similarly to open-telemetry#43117, the goleak errors occur because `regex.Parser` does not stop. Stopping a parser silences the error. Applied table-driven approach as the related benchmarks look very similar.  #### Link to tracking issue Fixes open-telemetry#43044  #### Testing ```sh cd pkg/stanza/operator/parser/regex go test -bench='Parser' ``` Co-authored-by: Andrzej Stencel <[email protected]>

#### Description A test was flaky due to depending on timing and an exact count of a periodic condition. By asserting instead that a minimum count was achieved instead, the problem can be sidestepped.  #### Link to tracking issue Fixes open-telemetry#42867  #### Testing `go test -count 100 -run TestStoreCredentials_PreexistingCredentialsAreUsed` will fail before this change is applied, and succeed after. Signed-off-by: Eric Chlebek <[email protected]>

…y#42963)  #### Description open-telemetry/opentelemetry-collector#13756 added support for exposing metrics' stability level in the generated documentation. This PR makes use of this functionality. We start by setting the stability of all metrics to `development`. More info about levels can be found at https://github.com/open-telemetry/opentelemetry-specification/blob/v1.49.0/oteps/0232-maturity-of-otel.md#maturity-levels. Related to: - open-telemetry/opentelemetry-collector#11878 - open-telemetry/opentelemetry-collector#13297 - open-telemetry#35325 - open-telemetry#42809  #### Link to tracking issue Fixes ~  #### Testing ~  #### Documentation Updated  Signed-off-by: ChrsMark <[email protected]>

#### Description open-telemetry/opentelemetry-collector#13756 added support for exposing metrics' stability level in the generated documentation. This PR makes use of this functionality. We start by setting the stability of all metrics to `development`. More info about levels can be found at https://github.com/open-telemetry/opentelemetry-specification/blob/v1.49.0/oteps/0232-maturity-of-otel.md#maturity-levels. Related to: - open-telemetry/opentelemetry-collector#11878 - open-telemetry/opentelemetry-collector#13297 - open-telemetry#35325 - open-telemetry#42809  #### Link to tracking issue Fixes ~  #### Testing ~  #### Documentation Updated  Signed-off-by: ChrsMark <[email protected]>

…open-telemetry#42935)  #### Description open-telemetry/opentelemetry-collector#13756 added support for exposing metrics' stability level in the generated documentation. This PR makes use of this functionality. We start by setting the stability of all metrics to `development`. More info about levels can be found at https://github.com/open-telemetry/opentelemetry-specification/blob/v1.49.0/oteps/0232-maturity-of-otel.md#maturity-levels. Related to: - open-telemetry/opentelemetry-collector#11878 - open-telemetry/opentelemetry-collector#13297 - open-telemetry#35325 - open-telemetry#42809  #### Link to tracking issue Fixes ~  #### Testing ~  #### Documentation Updated  Signed-off-by: ChrsMark <[email protected]>

#### Description This is another fix of open-telemetry#43044. Probably one more PR will be needed! Addressed 2 of the benchmark failures in `pkg/stanza/adapter` here. ``` --- FAIL: BenchmarkReadLine receiver_test.go:285: Error Trace: /Users/ray.kang/github.com/gnak-yar/opentelemetry-collector-contrib/pkg/stanza/adapter/receiver_test.go:285 Error: Received unexpected error: '' unsupported type 'file_input' Test: BenchmarkReadLine --- FAIL: BenchmarkParseAndMap receiver_test.go:369: Error Trace: /Users/ray.kang/github.com/gnak-yar/opentelemetry-collector-contrib/pkg/stanza/adapter/receiver_test.go:369 Error: Received unexpected error: unsupported type 'file_input' Test: BenchmarkParseAndMap ``` **BenchmarkReadLine** The `file_input` operator should be registered to a global operator registry before run. The registration is done when the file input operator package is imported, but the package isn't imported so the error occurs. I've removed the benchmark instead of fixing it. Here's the reason. - As far as I understand, this benchmark is to see the `file_input` operator's performance, but we have other benchmarks doing almost the same thing. - [file.BenchmarkReadExistingLogs](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/7e4bf11279e5b454980626237ccd1417b7e0e26a/pkg/stanza/operator/input/file/benchmark_test.go#L23) - [fileconsumer.BenchmarkFileInput](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/7e4bf11279e5b454980626237ccd1417b7e0e26a/pkg/stanza/fileconsumer/benchmark_test.go#L31) - Even the test seems a bit weird to me because the `b.N` loop(`for i := 0; i < b.N; i++ {`) is used for log generation, not for a function to be measured. **BenchmarkParseAndMap** Two reasons of the failure - The `file_input` package is not imported as well. - Unmarshaling yaml configs(line 369 before change) does not work well, as the configs are not supposed to be loaded from yaml directly. For example, the following structure is a part of `fileconsumer.Config`. `yaml.Unmarshaler` does not understand the embeded fields by default(Probably a more tag like `yaml:",inline"` is required). ```go type Config struct { matcher.Criteria `mapstructure:",squash"` attrs.Resolver `mapstructure:",squash"` ... } ``` As far as I understand, `BenchmarkParseAndMap` is to see the performance of `regex_parser`'s severity mappings. So, I've moved the benchmark to `regex_parser` and made it focus on the operator's performance. I'm not familiar with the full history and background. Please let me know if you have any concerns or suggestions!  #### Link to tracking issue Fixes open-telemetry#43044  #### Testing ``` cd pkg/stanza/adapter go test -bench='^Benchmark(ReadLine|ParseAndMap)$' # For testing a benchmark moved to the regex parser. cd pkg/stanza/operator/parser/regex go test -bench='^BenchmarkProcessBatch' ``` --------- Co-authored-by: Andrzej Stencel <[email protected]>

…lemetry#42989)  #### Description open-telemetry/opentelemetry-collector#13756 added support for exposing metrics' stability level in the generated documentation. This PR makes use of this functionality. We start by setting the stability of all metrics to `development`. More info about levels can be found at https://github.com/open-telemetry/opentelemetry-specification/blob/v1.49.0/oteps/0232-maturity-of-otel.md#maturity-levels. Related to: - open-telemetry/opentelemetry-collector#11878 - open-telemetry/opentelemetry-collector#13297 - open-telemetry#35325 - open-telemetry#42809  #### Link to tracking issue Fixes ~  #### Testing ~  #### Documentation Updated  Signed-off-by: ChrsMark <[email protected]>

…try#42951)  #### Description open-telemetry/opentelemetry-collector#13756 added support for exposing metrics' stability level in the generated documentation. This PR makes use of this functionality. We start by setting the stability of all metrics to `development`. More info about levels can be found at https://github.com/open-telemetry/opentelemetry-specification/blob/v1.49.0/oteps/0232-maturity-of-otel.md#maturity-levels. Related to: - open-telemetry/opentelemetry-collector#11878 - open-telemetry/opentelemetry-collector#13297 - open-telemetry#35325 - open-telemetry#42809  #### Link to tracking issue Fixes ~  #### Testing ~  #### Documentation Updated  Signed-off-by: ChrsMark <[email protected]>

#### Description downgrade datadog-go/v5 library to previous commit hash (no changes to user/api)  #### Link to tracking issue Fixes open-telemetry#43399  #### Testing existing testing  #### Documentation none

#### Description This PR introduced query-level data collection for `mysql` receiver in the logs pipeline. We introduced `Top Query` collection in this initial PR (`Top Queries` are those queries which used the most time within a time window) ##### Configuration We introduced these four configurations for the feature(see receiver's README for details): 1. `max_sample_query_count`: the initial query count to fetch from database. 2. `top_query_count`: the number to report to the next consumer. 3. `lookback_time`: the query window for each scrape. 4. `collection_interval`: the query interval for top collection collection. 5. `query_plan_cache_size`: the query plan cache size. 6. `query_plan_cache_ttl`: the query plan cache ttl time. ##### Workflow The `mysql` receiver will fetch M(=`max_sample_query_count`) queries from database and sort the queries according to the difference of `sum_timer_wait`(time used), and then report the first N(=`top_query_count`) queries. ##### New Log Attributes - `db.system.name` - `db.query.text` - `mysql.query_plan` - `mysql.events_statements_summary_by_digest.digest` - `mysql.events_statements_summary_by_digest.count_star` - `mysql.events_statements_summary_by_digest.sum_timer_wait` ##### Additional dependency * `hashicorp/golang-lru/v2` * License: MPL-2.0 * Link: https://pkg.go.dev/github.com/hashicorp/golang-lru/v2 * Already been used in the repo * `DataDog/datadog-agent/pkg/obfuscate` * License: Apache 2.0 * Link: https://pkg.go.dev/github.com/DataDog/datadog-agent/pkg/obfuscate * Already been used in the repo ##### Example Output ``` resourceLogs: - resource: {} scopeLogs: - logRecords: - attributes: - key: db.system.name value: stringValue: mysql - key: db.query.text value: stringValue: SELECT COALESCE ( schema_name, ? ) COALESCE ( digest, ? ) COALESCE ( digest_text, ? ) count_star, sum_timer_wait, query_sample_text FROM performance_schema.events_statements_summary_by_digest WHERE last_seen >= NOW ( ) - INTERVAL ? second AND ( ( digest_text NOT LIKE ? AND digest_text NOT LIKE ? ) OR digest_text IS ? ) ORDER BY count_star DESC LIMIT ? - key: mysql.query_plan value: stringValue: | {"query_block":{"select_id":1,"cost_info":{"query_cost":"?"},"ordering_operation":{"using_filesort":true,"table":{"table_name":"events_statements_summary_by_digest","access_type":"ALL","rows_examined_per_scan":"?","rows_produced_per_join":"?","filtered":"?","cost_info":{"read_cost":"?","eval_cost":"?","prefix_cost":"?","data_read_per_join":"?"},"used_columns":["SCHEMA_NAME","DIGEST","DIGEST_TEXT","COUNT_STAR","SUM_TIMER_WAIT","LAST_SEEN","QUERY_SAMPLE_TEXT"],"attached_condition":"( ( performance_schema . events_statements_summary_by_digest . LAST_SEEN >= < cache > ( ( now ( ) - interval ? second ) ) ) and ( ( ( not ( ( performance_schema . events_statements_summary_by_digest . DIGEST_TEXT like ? ) ) ) and ( not ( ( performance_schema . events_statements_summary_by_digest . DIGEST_TEXT like ? ) ) ) ) or ( performance_schema . events_statements_summary_by_digest . DIGEST_TEXT is ? ) ) )"}}}} - key: mysql.events_statements_summary_by_digest.digest value: stringValue: c16f24f908846019a741db580f6545a5933e9435a7cf1579c50794a6ca287739 - key: mysql.events_statements_summary_by_digest.count_star value: intValue: "5" - key: mysql.events_statements_summary_by_digest.sum_timer_wait value: doubleValue: 0.001021918999 body: {} eventName: db.server.top_query timeUnixNano: "1754297675177556000" scope: name: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/mysqlreceiver version: latest ```  #### Link to tracking issue n/a  #### Testing Added  #### Documentation Updated  --------- Co-authored-by: Nico Stewart <[email protected]> Co-authored-by: Curtis Robert <[email protected]> Co-authored-by: Antoine Toulme <[email protected]> Co-authored-by: Antoine Toulme <[email protected]>

#### Description - Add templates file. - Add a template for the triager proposal PR  #### Link to tracking issue Fixes open-telemetry#43103

@atoulme

#43393)  #### Description Adding a new ICMP check receiver to perform echo requests to specified targets and collect metrics from responses. This is first PR that introduces basic structure only, following `CONTRIBUTING/Adding New Components` guidelines: - README.md with In Development stability - config.go with validation for ping/echo targets - factory.go using helper factory APIs (returns a no-op metrics receiver for lifecycle tests) - doc.go with `//go:generate mdatagen metadata.yaml` - metadata.yaml and generated code under internal/metadata/ - Basic unit tests (config, factory, lifecycle)  #### Link to tracking issue [#29009](#29009)  #### Testing - Unit tests: - Config validation tests - Factory defaults and lifecycle - Generated lifecycle tests - Local repository checks run: - `make checkdoc`, `make checkmetadata`, `make checkapi` - `make goporto`, `make crosslink`, `make gotidy` - `make genotelcontribcol`, `make genoteltestbedcol`, `make generate`  #### Documentation - README.md with generated status badge - metadata.yaml with development stability - doc.go enabling `mdatagen` code generation @atoulme - Component sponsor

#43407) …f-profiler to v0.0.202540  #### Description Superseds #42590.  #### Link to tracking issue Fixes  #### Testing  #### Documentation  --------- Signed-off-by: Florian Lehner <[email protected]>

… Batch API mode (#43047)  #### Description This PR revives the previously non-functional #40344, aiming to enhance performance by running metrics collection across Azure subscriptions in parallel. This significantly reduces latency when dealing with multiple subscriptions. Compared to the original PR, this version addresses a critical concurrency issue: ```fatal error: concurrent map read and map write``` Reference: https://victoriametrics.com/blog/go-sync-map/ To resolve this, I evaluated multiple approaches and found that using the concurrent-map library yielded the best results. Details of the evaluation and benchmarks are documented in concurrency_bench_report.md.  #### Link to tracking issue Fixes #39417  #### Testing During testing, a race condition was discovered in the Azure SDK related to the fallback behavior of the Cloud option in the NewClient function: ```go // NewClient creates a client that accesses Azure Monitor metrics data. // Client should be used for performing metrics queries on multiple monitored resources in the same region. // A credential with authorization at the subscription level is required when using this client. // // endpoint - The regional endpoint to use, for example https://eastus.metrics.monitor.azure.com. // The region should match the region of the requested resources. For global resources, the region should be 'global'. func NewClient(endpoint string, credential azcore.TokenCredential, options *ClientOptions) (*Client, error) { if options == nil { options = &ClientOptions{} } if reflect.ValueOf(options.Cloud).IsZero() { options.Cloud = cloud.AzurePublic // <-- HERE } c, ok := options.Cloud.Services[ServiceName] if !ok || c.Audience == "" { return nil, errors.New("provided Cloud field is missing Azure Monitor Metrics configuration") } ``` To prevent this, our implementation explicitly sets the Cloud option in all cases, ensuring deterministic behavior and avoiding the race.  #### Documentation A new markdown file (concurrency_bench_report.md) has been added to document: - The rationale behind choosing concurrent-map - Benchmark results comparing different implementations - Notes for future contributors who may want to explore alternative concurrency strategies  --------- Signed-off-by: Célian Garcia <[email protected]> Signed-off-by: Celian GARCIA <[email protected]> Co-authored-by: Moritz Wiesinger <[email protected]>

#### Description upgrades datadog-go/v5 library to v5.8.1  #### Link to tracking issue Follow-up to #43399 after DataDog/datadog-go#340 was closed  #### Testing existing testing  #### Documentation none, dependency upgrade

#### Description removes `logs::dump_payloads` config from `pkg/datadog` (removing the option from datadogexporter)  #### Link to tracking issue Fixes #43427  #### Testing none, as functionality was already removed  #### Documentation breaking user changelog, deprecation api changelog

following up #34908 #34627  #### Description  #### Link to tracking issue Fixes  #### Testing  #### Documentation  Signed-off-by: Jared Tan <[email protected]>

jpkrohling requested a review from a team October 9, 2020 14:04

bogdandrutu reviewed Oct 12, 2020

View reviewed changes

README.md Outdated Show resolved Hide resolved

bogdandrutu changed the title ~~Added section on how to build an otelcol dist~~ Add section on how to build an otelcol dist Oct 12, 2020

jpkrohling added 2 commits October 12, 2020 17:31

Added section on how to build an otelcol dist

3166666

Signed-off-by: Juraci Paixão Kröhling <[email protected]>

s/OpenTracing/OpenTelemetry

acf4d51

Signed-off-by: Juraci Paixão Kröhling <[email protected]>

jpkrohling force-pushed the jpkrohling/add-link-to-blog-post-on-building-custom-otelcol branch from 73f9801 to acf4d51 Compare October 12, 2020 15:31

owais reviewed Oct 12, 2020

View reviewed changes

anuraaga reviewed Oct 13, 2020

View reviewed changes

README.md Show resolved Hide resolved

owais approved these changes Oct 13, 2020

View reviewed changes

anuraaga approved these changes Oct 14, 2020

View reviewed changes

bogdandrutu merged commit a987792 into open-telemetry:master Oct 14, 2020

kisieland referenced this pull request in kisieland/opentelemetry-collector-contrib Mar 16, 2021

Use cached tools instead of uploading to the artifacts (#2475)

9b2660f

Updates #1234 Signed-off-by: Bogdan Drutu <[email protected]>

ljmsc referenced this pull request in ljmsc/opentelemetry-collector-contrib Feb 21, 2022

Move asyncronous metrics state helper out of apimetric/metrictes (#1234)

f60f51d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add section on how to build an otelcol dist #1234

Add section on how to build an otelcol dist #1234

Uh oh!

jpkrohling commented Oct 9, 2020

Uh oh!

Uh oh!

owais Oct 12, 2020 •

edited

Loading

Uh oh!

owais Oct 12, 2020

Uh oh!

jpkrohling Oct 12, 2020 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Add section on how to build an otelcol dist #1234

Add section on how to build an otelcol dist #1234

Uh oh!

Conversation

jpkrohling commented Oct 9, 2020

Uh oh!

Uh oh!

owais Oct 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

owais Oct 12, 2020

Choose a reason for hiding this comment

Uh oh!

jpkrohling Oct 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

owais Oct 12, 2020 •

edited

Loading

jpkrohling Oct 12, 2020 •

edited

Loading