Thank you to the Prometheus team for the work on PR #17387
adding the new type-and-unit-labels support for Remote Write v2.

I noticed that this PR has been merged recently and decided to test the feature in practice.
However, during testing I found that __type__ and __unit__ labels do not appear on the Prometheus Receiver side when metrics are sent via remote_write v2 from Prometheus Agent - even when feature flags are enabled on both ends.

This behavior seems to contradict expectations, and might indicate either a missing configuration detail or a remaining issue in handling metadata through remote write v2.

Important note:
I previously opened an issue regarding this behavior, and it was closed as resolved. However, after further >testing, the problem still persists.
Therefore I am reopening the discussion here with a minimal reproducible configuration below.

Setup:

Prometheus (Receiver) startup flags:

--config.file=/opt/prometheus/prometheus.yml
--storage.tsdb.path=/mnt/tsdb/prometheus
--web.listen-address=127.0.0.1:9090
--web.enable-remote-write-receiver
--web.remote-write-receiver.accepted-protobuf-messages=io.prometheus.write.v2.Request
--enable-feature=type-and-unit-labels,metadata-wal-records
--log.level=debug

Prometheus (Agent) startup flags:

--config.file=/opt/prometheus-agent/prometheus-agent.yml
--agent
--web.listen-address=127.0.0.1:9091
--storage.agent.path=/mnt/tsdb/prometheus-agent
--enable-feature=type-and-unit-labels,metadata-wal-records
--log.level=debug

Agent configuration

global:
  scrape_interval: 15s
  scrape_timeout: 10s

scrape_configs:
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['test-node-exporter:9100']
  - job_name: 'random-data'
    static_configs:
      - targets: ['random-data-exporter:8081']

remote_write:
  - url: "http://127.0.0.1:9090/api/v1/write"
    name: "Remote write to prometheus"
    protobuf_message: io.prometheus.write.v2.Request

Observed behavior

Querying the receiver returns metrics without type and unit labels:

❯ curl -s 'http://127.0.0.1:9090/api/v1/query?query=node_cpu_seconds_total'  | jq '.data.result[0].metric'

{
  "__name__": "node_cpu_seconds_total",
  "cpu": "0",
  "instance": "test-node-exporter",
  "job": "node-exporter",
  "mode": "idle",
  "port": "9100"
}

Expected behavior:

{
  "__name__": "node_cpu_seconds_total",
  "__type__": "counter",
  "__unit__": "seconds",
  ...
}

Environment

• Prometheus version: version 3.7.3 (branch: main, revision: 03d0c18)
• Mode: Prometheus Receiver + Prometheus Agent
• OS: Alma Linux 9.6 (Sage Margay)
• Architecture: x86_64

If I am missing an additional step needed to propagate metadata via remote_write v2, please let me know.

Thank you in advance!