Skip to content

Conversation

SpencerTorres
Copy link
Member

@SpencerTorres SpencerTorres commented Jun 9, 2025

Description

Depends on refactoring from #40536

Adds JSON support to the exporter.

Summary:

  • Added feature gate (clickhouse.json, alpha stability) for enabling the JSON type pipeline. The Map based pipeline is still functional and is the default. (How to use feature gates)
  • The JSON type replaces all Map columns with the new JSON type.
  • JSON is sent to the server as a string.
  • Due to how the server stores JSON paths, OTel standard paths such as a.b.c are returned as objects a { b { c } } by the server. This can be changed via server setting in the future to return all data as flattened paths a.b.c.

Let me know if you have any questions, suggestions, etc.

Thanks!

Testing

  • Added unit tests for related helper functions
  • Added integration tests for JSON logs/traces

Documentation

  • Updated README, changelog

@SpencerTorres SpencerTorres requested a review from dmitryax as a code owner June 9, 2025 06:46
@SpencerTorres
Copy link
Member Author

SpencerTorres commented Jun 17, 2025

This PR includes a driver update that fixes the HTTP pipelines, if we don't get this merged before the next release I can submit a separate PR to update the module.

clickhouse-go v2.37.1 includes the fix, main is currently on v2.37.0

Without the fix, HTTP pipelines risk starving connections and inconsistent HTTP/400 errors.

Resolved in #40766

@github-actions github-actions bot requested a review from Frapschen June 22, 2025 00:39
@SpencerTorres SpencerTorres requested a review from dmitryax June 22, 2025 00:44
@dmitryax dmitryax merged commit b8746a9 into open-telemetry:main Jun 27, 2025
177 checks passed
@github-actions github-actions bot added this to the next release milestone Jun 27, 2025
@SpencerTorres SpencerTorres deleted the clickhouse_json_feature branch June 27, 2025 18:10
AvihuHenya added a commit to odigos-io/odigos that referenced this pull request Jul 31, 2025
…g for metrics tables (#3206)

## Description

This PR brings two main enhancements:

### 1. **Support for ClickHouse JSON Column Type**

- Applies the upstream update from
[open-telemetry-collector-contrib#40547](open-telemetry/opentelemetry-collector-contrib#40547)
to enable support for the `JSON` column type in ClickHouse (available as
of version `v25.3`).
- Enables appending the `enable_json_type=1` flag to the ClickHouse
exporter endpoint via the destination configuration.
- This allows telemetry data to include JSON-typed fields when exporting
to ClickHouse.

### 2. **Configurable Metric Table Names**

- Adds support for customizing metrics table names through the
destination configuration, giving users more control over their schema
design.

### 3. **Feature Gate for JSON Support**

- Introduces a feature gate for enabling JSON support in ClickHouse via
the Helm chart and CLI.
- Adds a new `config set` field for injecting
`--feature-gates=clickhouse.json` as an argument to the gateway. This
change tells the gateway to push telemetry data to clickhouse using JSON
column types. The gateway will only export telemetry data using JSON
column types when this feature gate is explicitly enabled.

### Docs
- Fixes a minor documentation issue: updates the `mint.json` reference
for the `odigos config` command.


## How Has This Been Tested?

<!-- Describe the tests you ran and how you verified your changes. -->

- [ ] Added Unit Tests
- [ ] Updated e2e Tests
- [ ] Manual Testing
- [ ] Manual Load Test

## Kubernetes Checklist

<!-- If this PR affects how Odigos interacts with Kubernetes, check the
relevant boxes below and provide more details -->

- [ ] Changes how Odigos interacts with Kubernetes
- [ ] Introduces additional calls to the API Server (potential
performance impact)
- [ ] New Query/feature supported in all the k8s versions supported by
Odigos
- [ ] Modifies Odigos manifests (addressed in both CLI and Helm)
- [ ] Changes RBAC permissions

## User Facing Changes

<!-- Any changes that users will notice or need to be aware of -->

- [ ] Users need to take action before upgrading
- [ ] Automatic migration will modify existing objects (backward
compatible)
- [ ] Changes UI, CLI, or K8s Manifests aspects in a way that users need
to be aware of
- [ ] Documentation updated accordingly

---------

Co-authored-by: Amir Blum <[email protected]>
Dylan-M pushed a commit to Dylan-M/opentelemetry-collector-contrib that referenced this pull request Aug 5, 2025
…pen-telemetry#40547)

#### Description

Depends on refactoring from open-telemetry#40536

Adds JSON support to the exporter.

Summary:
- Added feature gate (`clickhouse.json`, alpha stability) for enabling
the JSON type pipeline. The `Map` based pipeline is still functional and
is the default. ([How to use feature
gates](https://github.com/open-telemetry/opentelemetry-collector/blob/main/featuregate/README.md#controlling-gates))
- The JSON type replaces all `Map` columns with the [new JSON
type](https://clickhouse.com/docs/sql-reference/data-types/newjson).
- JSON is sent to the server as a string.
- Due to how the server stores JSON paths, OTel standard paths such as
`a.b.c` are returned as objects `a { b { c } }` by the server. This can
be changed via server setting in the future to return all data as
flattened paths `a.b.c`.

Let me know if you have any questions, suggestions, etc.

Thanks!

#### Testing

- Added unit tests for related helper functions
- Added integration tests for JSON logs/traces

#### Documentation

- Updated README, changelog
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants