-
Notifications
You must be signed in to change notification settings - Fork 3.1k
[chore] [exporter/clickhouseexporter] full refactor for ClickHouse component #40536
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
dmitryax
merged 30 commits into
open-telemetry:main
from
SpencerTorres:clickhouse_full_reorganize
Jun 16, 2025
Merged
[chore] [exporter/clickhouseexporter] full refactor for ClickHouse component #40536
dmitryax
merged 30 commits into
open-telemetry:main
from
SpencerTorres:clickhouse_full_reorganize
Jun 16, 2025
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…tor-contrib into clickhouse_full_reorganize
Great. |
hanjm
approved these changes
Jun 8, 2025
This was referenced Jun 9, 2025
…tor-contrib into clickhouse_full_reorganize
…tor-contrib into clickhouse_full_reorganize
dmitryax
approved these changes
Jun 16, 2025
BenElferink
added a commit
to odigos-io/odigos
that referenced
this pull request
Jun 17, 2025
## Description The following PR broke the Clickhouse default DDL link: open-telemetry/opentelemetry-collector-contrib#40536 This PR locks the link at a specific tree to prevent 404 not found. Also, the following broken links were fixed: - MariaDB MySqlConnector for ADO.NET
This was referenced Jun 18, 2025
dmitryax
pushed a commit
that referenced
this pull request
Jun 27, 2025
…40547) #### Description Depends on refactoring from #40536 Adds JSON support to the exporter. Summary: - Added feature gate (`clickhouse.json`, alpha stability) for enabling the JSON type pipeline. The `Map` based pipeline is still functional and is the default. ([How to use feature gates](https://github.com/open-telemetry/opentelemetry-collector/blob/main/featuregate/README.md#controlling-gates)) - The JSON type replaces all `Map` columns with the [new JSON type](https://clickhouse.com/docs/sql-reference/data-types/newjson). - JSON is sent to the server as a string. - Due to how the server stores JSON paths, OTel standard paths such as `a.b.c` are returned as objects `a { b { c } }` by the server. This can be changed via server setting in the future to return all data as flattened paths `a.b.c`. Let me know if you have any questions, suggestions, etc. Thanks! #### Testing - Added unit tests for related helper functions - Added integration tests for JSON logs/traces #### Documentation - Updated README, changelog
Dylan-M
pushed a commit
to Dylan-M/opentelemetry-collector-contrib
that referenced
this pull request
Aug 5, 2025
…mponent (open-telemetry#40536) #### Description This PR intends to reorganize the code of the exporter without making any user-facing changes to the behavior or config. The goal is to make it more maintainable and revise all previous changes into one cohesive component. Apologies in advance for the many changes, but I wanted this to be runnable so we can merge it into main. **Let me know if we need to split anything out, or make any changes.** Notable changes: - All `clickhouse-go` database calls now use the library's native Go interface instead of the `database/sql` interface. This is more efficient and easier to configure + test. There was a lot of overhead in converting from `database/sql` to the driver's native structures, so this has a performance benefit as well as more idiomatic code. - Files have been reorganized so that code is in a more logical and easy to find place. (For example, there were some shared functions inside the `exporter_logs.go` file, even though it was unrelated to logs) - SQL is no longer written inside the `.go` file, and is now using `go:embed` to embed it from a directory of files. - Moved generic code to the `internal` package to simplify the top level package's exports. - Optimized insert code for logs/traces to be more CPU efficient (there were some redundant calls for converting attributes to `map`, wasting CPU). See testing section below for notes on test changes. #### Testing - Mocked database tests were replaced with full integration tests for logs, traces, and metrics. - Integration test share the same test container, ensuring quicker testing (no more starting several containers for simple query tests) - Preserved `goleak` checks for integration tests (and unit tests) - Integration tests were split into multiple files (these are in `*_integration_test.go` files) - Unit tests are separate from the integration tests (these are in the usual `_test.go` files) - Added tests for new functions/cases - Removed irrelevant/unhelpful tests
Dylan-M
pushed a commit
to Dylan-M/opentelemetry-collector-contrib
that referenced
this pull request
Aug 5, 2025
…pen-telemetry#40547) #### Description Depends on refactoring from open-telemetry#40536 Adds JSON support to the exporter. Summary: - Added feature gate (`clickhouse.json`, alpha stability) for enabling the JSON type pipeline. The `Map` based pipeline is still functional and is the default. ([How to use feature gates](https://github.com/open-telemetry/opentelemetry-collector/blob/main/featuregate/README.md#controlling-gates)) - The JSON type replaces all `Map` columns with the [new JSON type](https://clickhouse.com/docs/sql-reference/data-types/newjson). - JSON is sent to the server as a string. - Due to how the server stores JSON paths, OTel standard paths such as `a.b.c` are returned as objects `a { b { c } }` by the server. This can be changed via server setting in the future to return all data as flattened paths `a.b.c`. Let me know if you have any questions, suggestions, etc. Thanks! #### Testing - Added unit tests for related helper functions - Added integration tests for JSON logs/traces #### Documentation - Updated README, changelog
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR intends to reorganize the code of the exporter without making any user-facing changes to the behavior or config. The goal is to make it more maintainable and revise all previous changes into one cohesive component.
Apologies in advance for the many changes, but I wanted this to be runnable so we can merge it into main. Let me know if we need to split anything out, or make any changes.
Notable changes:
clickhouse-go
database calls now use the library's native Go interface instead of thedatabase/sql
interface. This is more efficient and easier to configure + test. There was a lot of overhead in converting fromdatabase/sql
to the driver's native structures, so this has a performance benefit as well as more idiomatic code.exporter_logs.go
file, even though it was unrelated to logs).go
file, and is now usinggo:embed
to embed it from a directory of files.internal
package to simplify the top level package's exports.map
, wasting CPU).See testing section below for notes on test changes.
Testing
goleak
checks for integration tests (and unit tests)*_integration_test.go
files)_test.go
files)