Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

prometheusremotewrite exporter pushes all metrics as Gauges #26568

Closed
JanPetr opened this issue Sep 11, 2023 · 3 comments
Closed

prometheusremotewrite exporter pushes all metrics as Gauges #26568

JanPetr opened this issue Sep 11, 2023 · 3 comments
Labels
bug Something isn't working closed as inactive data:metrics Metric related issues exporter/prometheusremotewrite Stale

Comments

@JanPetr
Copy link

JanPetr commented Sep 11, 2023 •
edited
Loading

Describe the bug
prometheusremotewrite exporter pushes all metrics as Gauges even though some metrics should be pushed as Counters.

Steps to reproduce
I have this OTel config.yaml:

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'otel-collector'
          scrape_interval: 10s
          static_configs:
            - targets: [ '0.0.0.0:8888' ]

processors:
  batch:

exporters:
  prometheusremotewrite:
    endpoint: "http://0.0.0.0:8999"

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [prometheusremotewrite]

I accept exported metrics via Vector with this config:

#[sources.prometheus_metrics]
#type = "prometheus_scrape"
#endpoints = [ "http://localhost:8888/metrics" ]

[sources.prometheus_metrics]
type = "prometheus_remote_write"
address = "0.0.0.0:8999"

[transforms.prometheus_metrics_to_log]
type = "metric_to_log"
inputs = ["prometheus_metrics"]

[transforms.prometheus_metrics_add_token]
type = "remap"
inputs = ["prometheus_metrics_to_log"]
source = '''
.dt = del(.timestamp)
.__source_token = del(.tags.source_token)
'''

[sinks.console]
type = "console"
inputs = ["prometheus_metrics_add_token"]
encoding.codec = "json"

What did you expect to see?
The output metrics are these:

{"__source_token":null,"dt":"2023-09-11T12:35:02.282Z","gauge":{"value":0.18},"kind":"absolute","name":"otelcol_process_cpu_seconds","tags":{"http_scheme":"http","instance":"0.0.0.0:8888","job":"otel-collector","net_host_port":"8888","service_instance_id":"0.0.0.0:8888;88911463-c5a5-4581-847a-d3c1438ebe0f","service_name":"otel-collector;otelcol-contrib","service_version":"0.84.0"}}
{"__source_token":null,"dt":"2023-09-11T12:35:02.282Z","gauge":{"value":0.015557791},"kind":"absolute","name":"scrape_duration_seconds","tags":{"http_scheme":"http","instance":"0.0.0.0:8888","job":"otel-collector","net_host_port":"8888","service_instance_id":"0.0.0.0:8888","service_name":"otel-collector"}}
{"__source_token":null,"dt":"2023-09-11T12:35:02.282Z","gauge":{"value":58361096.0},"kind":"absolute","name":"otelcol_process_runtime_total_sys_memory_bytes","tags":{"http_scheme":"http","instance":"0.0.0.0:8888","job":"otel-collector","net_host_port":"8888","service_instance_id":"0.0.0.0:8888;88911463-c5a5-4581-847a-d3c1438ebe0f","service_name":"otel-collector;otelcol-contrib","service_version":"0.84.0"}}
{"__source_token":null,"dt":"2023-09-11T12:35:02.282Z","gauge":{"value":5.779557},"kind":"absolute","name":"otelcol_process_uptime","tags":{"http_scheme":"http","instance":"0.0.0.0:8888","job":"otel-collector","net_host_port":"8888","service_instance_id":"0.0.0.0:8888;88911463-c5a5-4581-847a-d3c1438ebe0f","service_name":"otel-collector;otelcol-contrib","service_version":"0.84.0"}}
{"__source_token":null,"dt":"2023-09-11T12:35:02.282Z","gauge":{"value":0.0},"kind":"absolute","name":"otelcol_exporter_enqueue_failed_log_records","tags":{"exporter":"prometheusremotewrite","http_scheme":"http","instance":"0.0.0.0:8888","job":"otel-collector","net_host_port":"8888","service_instance_id":"0.0.0.0:8888;88911463-c5a5-4581-847a-d3c1438ebe0f","service_name":"otel-collector;otelcol-contrib","service_version":"0.84.0"}}
{"__source_token":null,"dt":"2023-09-11T12:35:02.282Z","gauge":{"value":1.0},"kind":"absolute","name":"target_info","tags":{"http_scheme":"http","instance":"0.0.0.0:8888","job":"otel-collector","net_host_port":"8888"}}
{"__source_token":null,"dt":"2023-09-11T12:35:02.282Z","gauge":{"value":133332992.0},"kind":"absolute","name":"otelcol_process_memory_rss","tags":{"http_scheme":"http","instance":"0.0.0.0:8888","job":"otel-collector","net_host_port":"8888","service_instance_id":"0.0.0.0:8888;88911463-c5a5-4581-847a-d3c1438ebe0f","service_name":"otel-collector;otelcol-contrib","service_version":"0.84.0"}}
{"__source_token":null,"dt":"2023-09-11T12:35:02.282Z","gauge":{"value":11.0},"kind":"absolute","name":"scrape_samples_scraped","tags":{"http_scheme":"http","instance":"0.0.0.0:8888","job":"otel-collector","net_host_port":"8888","service_instance_id":"0.0.0.0:8888","service_name":"otel-collector"}}
{"__source_token":null,"dt":"2023-09-11T12:35:02.282Z","gauge":{"value":0.0},"kind":"absolute","name":"otelcol_exporter_enqueue_failed_spans","tags":{"exporter":"prometheusremotewrite","http_scheme":"http","instance":"0.0.0.0:8888","job":"otel-collector","net_host_port":"8888","service_instance_id":"0.0.0.0:8888;88911463-c5a5-4581-847a-d3c1438ebe0f","service_name":"otel-collector;otelcol-contrib","service_version":"0.84.0"}}
{"__source_token":null,"dt":"2023-09-11T12:35:02.282Z","gauge":{"value":11.0},"kind":"absolute","name":"scrape_series_added","tags":{"http_scheme":"http","instance":"0.0.0.0:8888","job":"otel-collector","net_host_port":"8888","service_instance_id":"0.0.0.0:8888","service_name":"otel-collector"}}
{"__source_token":null,"dt":"2023-09-11T12:35:02.282Z","gauge":{"value":0.0},"kind":"absolute","name":"otelcol_exporter_queue_size","tags":{"exporter":"prometheusremotewrite","http_scheme":"http","instance":"0.0.0.0:8888","job":"otel-collector","net_host_port":"8888","service_instance_id":"0.0.0.0:8888;88911463-c5a5-4581-847a-d3c1438ebe0f","service_name":"otel-collector;otelcol-contrib","service_version":"0.84.0"}}
{"__source_token":null,"dt":"2023-09-11T12:35:02.282Z","gauge":{"value":10000.0},"kind":"absolute","name":"otelcol_exporter_queue_capacity","tags":{"exporter":"prometheusremotewrite","http_scheme":"http","instance":"0.0.0.0:8888","job":"otel-collector","net_host_port":"8888","service_instance_id":"0.0.0.0:8888;88911463-c5a5-4581-847a-d3c1438ebe0f","service_name":"otel-collector;otelcol-contrib","service_version":"0.84.0"}}
{"__source_token":null,"dt":"2023-09-11T12:35:02.282Z","gauge":{"value":47306544.0},"kind":"absolute","name":"otelcol_process_runtime_total_alloc_bytes","tags":{"http_scheme":"http","instance":"0.0.0.0:8888","job":"otel-collector","net_host_port":"8888","service_instance_id":"0.0.0.0:8888;88911463-c5a5-4581-847a-d3c1438ebe0f","service_name":"otel-collector;otelcol-contrib","service_version":"0.84.0"}}
{"__source_token":null,"dt":"2023-09-11T12:35:02.282Z","gauge":{"value":11.0},"kind":"absolute","name":"scrape_samples_post_metric_relabeling","tags":{"http_scheme":"http","instance":"0.0.0.0:8888","job":"otel-collector","net_host_port":"8888","service_instance_id":"0.0.0.0:8888","service_name":"otel-collector"}}
{"__source_token":null,"dt":"2023-09-11T12:35:02.282Z","gauge":{"value":43527456.0},"kind":"absolute","name":"otelcol_process_runtime_heap_alloc_bytes","tags":{"http_scheme":"http","instance":"0.0.0.0:8888","job":"otel-collector","net_host_port":"8888","service_instance_id":"0.0.0.0:8888;88911463-c5a5-4581-847a-d3c1438ebe0f","service_name":"otel-collector;otelcol-contrib","service_version":"0.84.0"}}
{"__source_token":null,"dt":"2023-09-11T12:35:02.282Z","gauge":{"value":1.0},"kind":"absolute","name":"up","tags":{"http_scheme":"http","instance":"0.0.0.0:8888","job":"otel-collector","net_host_port":"8888","service_instance_id":"0.0.0.0:8888","service_name":"otel-collector"}}
{"__source_token":null,"dt":"2023-09-11T12:35:02.282Z","gauge":{"value":0.0},"kind":"absolute","name":"otelcol_exporter_enqueue_failed_metric_points","tags":{"exporter":"prometheusremotewrite","http_scheme":"http","instance":"0.0.0.0:8888","job":"otel-collector","net_host_port":"8888","service_instance_id":"0.0.0.0:8888;88911463-c5a5-4581-847a-d3c1438ebe0f","service_name":"otel-collector;otelcol-contrib","service_version":"0.84.0"}}

What did you see instead?
Counter Metrics reported as Counters:

{"__source_token":null,"counter":{"value":0.0},"dt":"2023-09-11T12:39:17.781483Z","kind":"absolute","name":"otelcol_exporter_enqueue_failed_log_records","tags":{"exporter":"prometheusremotewrite","service_instance_id":"88911463-c5a5-4581-847a-d3c1438ebe0f","service_name":"otelcol-contrib","service_version":"0.84.0"}}
{"__source_token":null,"counter":{"value":0.0},"dt":"2023-09-11T12:39:17.781483Z","kind":"absolute","name":"otelcol_exporter_enqueue_failed_metric_points","tags":{"exporter":"prometheusremotewrite","service_instance_id":"88911463-c5a5-4581-847a-d3c1438ebe0f","service_name":"otelcol-contrib","service_version":"0.84.0"}}
{"__source_token":null,"counter":{"value":0.0},"dt":"2023-09-11T12:39:17.781483Z","kind":"absolute","name":"otelcol_exporter_enqueue_failed_spans","tags":{"exporter":"prometheusremotewrite","service_instance_id":"88911463-c5a5-4581-847a-d3c1438ebe0f","service_name":"otelcol-contrib","service_version":"0.84.0"}}
{"__source_token":null,"dt":"2023-09-11T12:39:17.781483Z","gauge":{"value":10000.0},"kind":"absolute","name":"otelcol_exporter_queue_capacity","tags":{"exporter":"prometheusremotewrite","service_instance_id":"88911463-c5a5-4581-847a-d3c1438ebe0f","service_name":"otelcol-contrib","service_version":"0.84.0"}}
{"__source_token":null,"dt":"2023-09-11T12:39:17.781483Z","gauge":{"value":20.0},"kind":"absolute","name":"otelcol_exporter_queue_size","tags":{"exporter":"prometheusremotewrite","service_instance_id":"88911463-c5a5-4581-847a-d3c1438ebe0f","service_name":"otelcol-contrib","service_version":"0.84.0"}}
{"__source_token":null,"counter":{"value":76.0},"dt":"2023-09-11T12:39:17.781483Z","kind":"absolute","name":"otelcol_exporter_send_failed_metric_points","tags":{"exporter":"prometheusremotewrite","service_instance_id":"88911463-c5a5-4581-847a-d3c1438ebe0f","service_name":"otelcol-contrib","service_version":"0.84.0"}}
{"__source_token":null,"counter":{"value":16.0},"dt":"2023-09-11T12:39:17.781483Z","kind":"absolute","name":"otelcol_exporter_sent_metric_points","tags":{"exporter":"prometheusremotewrite","service_instance_id":"88911463-c5a5-4581-847a-d3c1438ebe0f","service_name":"otelcol-contrib","service_version":"0.84.0"}}
{"__source_token":null,"counter":{"value":1.19},"dt":"2023-09-11T12:39:17.781483Z","kind":"absolute","name":"otelcol_process_cpu_seconds","tags":{"service_instance_id":"88911463-c5a5-4581-847a-d3c1438ebe0f","service_name":"otelcol-contrib","service_version":"0.84.0"}}
{"__source_token":null,"dt":"2023-09-11T12:39:17.781483Z","gauge":{"value":146178048.0},"kind":"absolute","name":"otelcol_process_memory_rss","tags":{"service_instance_id":"88911463-c5a5-4581-847a-d3c1438ebe0f","service_name":"otelcol-contrib","service_version":"0.84.0"}}
{"__source_token":null,"dt":"2023-09-11T12:39:17.781483Z","gauge":{"value":51867968.0},"kind":"absolute","name":"otelcol_process_runtime_heap_alloc_bytes","tags":{"service_instance_id":"88911463-c5a5-4581-847a-d3c1438ebe0f","service_name":"otelcol-contrib","service_version":"0.84.0"}}
{"__source_token":null,"counter":{"value":87971072.0},"dt":"2023-09-11T12:39:17.781483Z","kind":"absolute","name":"otelcol_process_runtime_total_alloc_bytes","tags":{"service_instance_id":"88911463-c5a5-4581-847a-d3c1438ebe0f","service_name":"otelcol-contrib","service_version":"0.84.0"}}
{"__source_token":null,"dt":"2023-09-11T12:39:17.781483Z","gauge":{"value":71468296.0},"kind":"absolute","name":"otelcol_process_runtime_total_sys_memory_bytes","tags":{"service_instance_id":"88911463-c5a5-4581-847a-d3c1438ebe0f","service_name":"otelcol-contrib","service_version":"0.84.0"}}
{"__source_token":null,"counter":{"value":261.268082},"dt":"2023-09-11T12:39:17.781483Z","kind":"absolute","name":"otelcol_process_uptime","tags":{"service_instance_id":"88911463-c5a5-4581-847a-d3c1438ebe0f","service_name":"otelcol-contrib","service_version":"0.84.0"}}
{"__source_token":null,"counter":{"value":509.0},"dt":"2023-09-11T12:39:17.781483Z","kind":"absolute","name":"otelcol_receiver_accepted_metric_points","tags":{"receiver":"prometheus","service_instance_id":"88911463-c5a5-4581-847a-d3c1438ebe0f","service_name":"otelcol-contrib","service_version":"0.84.0","transport":"http"}}
{"__source_token":null,"counter":{"value":0.0},"dt":"2023-09-11T12:39:17.781483Z","kind":"absolute","name":"otelcol_receiver_refused_metric_points","tags":{"receiver":"prometheus","service_instance_id":"88911463-c5a5-4581-847a-d3c1438ebe0f","service_name":"otelcol-contrib","service_version":"0.84.0","transport":"http"}}

It works OK when I switch Vector source from prometheus_remote_write to prometheus_scrape to scrape Prometheus metrics endpoint directly (commented out in Vector config).

What version did you use?
Version: v0.84.0

What config did you use?
Config: (e.g. the yaml config file)

Environment
OS: MacOS 13.5.1 (22G90)
./otelcol-contrib downloaded directly from distro repo

Additional context
I tried to add additional configuration to prometheusremotewrite exporter, but didn't work either.

    resource_to_telemetry_conversion:
      enabled: true
    target_info:
      enabled: true
    add_metric_suffixes:
      true
@JanPetr JanPetr added the bug Something isn't working label Sep 11, 2023
@mx-psi mx-psi transferred this issue from open-telemetry/opentelemetry-collector Sep 11, 2023
@github-actions
Copy link
Contributor

Pinging code owners for exporter/prometheusremotewrite: @Aneurysm9 @rapphil. See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot mentioned this issue Sep 12, 2023
Open
This was referenced Sep 12, 2023
Open
Open
@github-actions GitHub Actions
Copy link
Contributor

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot added the Stale label Nov 13, 2023
@github-actions GitHub Actions
Copy link
Contributor

This issue has been closed as inactive because it has been stale for 120 days with no activity.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Jan 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working closed as inactive data:metrics Metric related issues exporter/prometheusremotewrite Stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@JanPetr @mx-psi