Accept data from datadog agent

[miki-sunlight]

Component(s)

receiver/datadog

Describe the issue you're reporting

Hey, trying to collect traces from DataDog agent using the OTEL collector with the DataDog Receiver.
On DataDog side I set up another end point by adding this code to the values.yaml file:

agents:
  useConfigMap: true
  customAgentConfig:
    apm_config:
      additional_endpoints:
        "http://otel-collector-cli-service:8126":
          - "1234" 

On the OTEL collector side I added the DataDog receiver:

    receivers:
      otlp:
        protocols:
          grpc:
          http:
      datadog:
        endpoint: 0.0.0.0:8126
        read_timeout: 60s
    service:
      telemetry:
        logs:
          level: "debug"
      pipelines:
        traces:
          receivers: [otlp, datadog]

When a trace is created I get the following warn on the DataDog trace agent:

trace-agent 2023-09-19 06:53:35 UTC | TRACE | WARN | (run.go:276 in Warnf) | Stats writer payload rejected by edge: 404 Not Found

The OTEL collector log show nothing.

If the OTEL collector is unreachable, The DataDog agent log this message:

trace-agent 2023-09-21 06:55:20 UTC | TRACE | WARN | (run.go:276 in Warnf) | Retried payload 4 times: Post "http://otel-collector-cli-service:8126/api/v0.2/traces": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Any idea what can be the problem? or if there is another way to collect traces from the DataDog agent?

[github-actions]

Pinging code owners:

• receiver/datadog: @boostchicken @gouthamve @jpkrohling @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[bar-alma]

BTW I reproduced the issue. Looks like agent version 7.47.0 tries to send data to /api/v2/series, but this path isn't defined in receiver.go.

No solution in sight.

[boostchicken]

A new format, yay/ Ill get something sorted

[boostchicken]

https://github.com/vectordotdev/vector/blob/master/src/sources/datadog_agent/traces.rs

[crobert-1]

Looks like a valid bug from comments, labeling as such and removing needs triage.

[karmingc]

@boostchicken is DD Agent -> datadogreceiver actually supported? My understanding is that it supports applications instrumented with DD client -> datadogreceiver directly.

edit: I should specify that I have limited understanding in Datadog and unsure if the format sent from the agent is different from the ones sent from the application instrumented with the Datadog tracer client.

[tomert-alma]

Hi, I made a fix in this commit #27672
can someone please review it?