[connector/routing] Routing by Attribute value fails and route to default pipeline #27020

nadavleva · 2023-09-20T12:33:13Z

Component(s)

connector/routing

What happened?

Description

I am running PoC to check routing by Resource Attributes using the connector routingconnector

I am trying to route to a different pipeline based on the Logs Severity Attribute

The routing by attribute fails to route to the correct attribute instead it routes to the default pipeline

The Logs output and otel-collector logs show the attribute value is "info" and the routing rule for info should be logs/apiengine exporter

Steps to Reproduce

Build customer otel collector including routingconnector, see
otelcol-builder.txt
Run the custom collector with the configuration
otel-collector-config_connector.txt
Copy a sample file with the severity attribute "info" to the path the collector file receiver is watching
input_sample.txt

Expected Result

The log resource should be routed to the pipeline logs/apiengine

Actual Result

The Log resource is routed to the default pipeline logs/all

Collector version

v0.85.0

Environment information

Environment

OS: Ubuntu 20.04.5 LTS
Compiler(if manually compiled): go1.21.1 linux/amd64

OpenTelemetry Collector configuration

receivers:
  filelog:
    include: [ /opt/parser/*.log ]
    start_at: beginning
    operators:
      - type: json_parser
        parse_from: body
        parse_to: attributes
      - type: severity_parser
        parse_from: attributes.sev
        mapping:
          fatal: emergency
          info4: notice
      - type: move
        id: move5
        from: attributes.sev
        to: attributes["Severity"]
      

exporters:
  file/file1:
    path: /var/log/collector/logging1/otlp.json
  file/file2:
    path: /var/log/collector/logging2/otlp.json
  file/file3:
    path: /var/log/collector/logging3/otlp.json
  logging:
    loglevel: debug


connectors:
  routing:
      default_pipelines: [logs/all]
      error_mode: propagate
      table:
        - statement: route() where attributes["Severity"] == "err"
          pipelines: [logs/apiengine]
        - statement: route() where attributes["Severity"] == "info"
          pipelines: [logs/msgq]

service:
  pipelines:
    logs/in:
      receivers: [filelog]
      exporters: [routing, logging]
    logs/all:
      receivers: [routing]
      exporters: [file/file3, logging]
    logs/msgq:
      receivers: [routing]
      exporters: [file/file1, logging]
    logs/apiengine:
      receivers: [routing]
      exporters: [file/file2, logging]

Log output

ut
2023-09-20T12:01:37.766Z	info	service/telemetry.go:84	Setting up own telemetry...
2023-09-20T12:01:37.767Z	info	service/telemetry.go:201	Serving Prometheus metrics	{"address": ":8888", "level": "Basic"}
2023-09-20T12:01:37.767Z	info	exporter@v0.85.0/exporter.go:275	Development component. May change in the future.	{"kind": "exporter", "data_type": "logs", "name": "logging"}
2023-09-20T12:01:37.767Z	warn	loggingexporter@v0.85.0/factory.go:98	'loglevel' option is deprecated in favor of 'verbosity'. Set 'verbosity' to equivalent value to preserve behavior.	{"kind": "exporter", "data_type": "logs", "name": "logging", "loglevel": "debug", "equivalent verbosity level": "Detailed"}
2023-09-20T12:01:37.770Z	info	connector@v0.85.0/connector.go:636	Development component. May change in the future.	{"kind": "exporter", "name": "routing", "exporter_in_pipeline": "logs", "receiver_in_pipeline": "logs"}
2023-09-20T12:01:37.772Z	info	service/service.go:138	Starting otelcol-custom...	{"Version": "1.0.0", "NumCPU": 8}
2023-09-20T12:01:37.772Z	info	extensions/extensions.go:31	Starting extensions...
2023-09-20T12:01:37.773Z	info	adapter/receiver.go:45	Starting stanza receiver	{"kind": "receiver", "name": "filelog", "data_type": "logs"}
2023-09-20T12:01:37.773Z	warn	fileconsumer/file.go:61	finding files: no files match the configured criteria	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2023-09-20T12:01:37.773Z	info	service/service.go:161	Everything is ready. Begin running and processing data.



2023-09-20T12:01:38.774Z	warn	fileconsumer/file.go:118	finding files: no files match the configured criteria	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}



2023-09-20T12:01:39.974Z	warn	fileconsumer/file.go:118	finding files: no files match the configured criteria	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2023-09-20T12:01:40.974Z	warn	fileconsumer/file.go:118	finding files: no files match the configured criteria	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2023-09-20T12:01:42.174Z	warn	fileconsumer/file.go:118	finding files: no files match the configured criteria	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2023-09-20T12:01:43.374Z	warn	fileconsumer/file.go:118	finding files: no files match the configured criteria	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2023-09-20T12:01:44.573Z	warn	fileconsumer/file.go:118	finding files: no files match the configured criteria	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2023-09-20T12:01:45.574Z	info	fileconsumer/file.go:194	Started watching file	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/opt/parser/t10.log"}
2023-09-20T12:01:45.673Z	info	LogsExporter	{"kind": "exporter", "data_type": "logs", "name": "logging", "resource logs": 1, "log records": 1}
2023-09-20T12:01:45.673Z	info	ResourceLog #0
Resource SchemaURL: 
ScopeLogs #0
ScopeLogs SchemaURL: 
InstrumentationScope  
LogRecord #0
ObservedTimestamp: 2023-09-20 12:01:45.574509522 +0000 UTC
Timestamp: 1970-01-01 00:00:00 +0000 UTC
SeverityText: info
SeverityNumber: Info(9)
Body: Str({"scid":"service-96km4711110", "sev":"info" ,"ct":"f5-api-service","ts":"2022-12-29 09:48:51.765624 UTC","log":{"time":"2022-12-29T09:48:51Z","log":"HEALTHY"}})
Attributes:
     -> log: Map({"log":"HEALTHY","time":"2022-12-29T09:48:51Z"})
     -> scid: Str(service-96km4711110)
     -> log.file.name: Str(t10.log)
     -> Severity: Str(info)
     -> ct: Str(f5-api-service)
     -> ts: Str(2022-12-29 09:48:51.765624 UTC)
Trace ID: 
Span ID: 
Flags: 0
	{"kind": "exporter", "data_type": "logs", "name": "logging"}
2023-09-20T12:01:45.675Z	info	LogsExporter	{"kind": "exporter", "data_type": "logs", "name": "logging", "resource logs": 1, "log records": 1}
2023-09-20T12:01:45.676Z	info	ResourceLog #0
Resource SchemaURL: 
ScopeLogs #0
ScopeLogs SchemaURL: 
InstrumentationScope  
LogRecord #0
ObservedTimestamp: 2023-09-20 12:01:45.574509522 +0000 UTC
Timestamp: 1970-01-01 00:00:00 +0000 UTC
SeverityText: info
SeverityNumber: Info(9)
Body: Str({"scid":"service-96km4711110", "sev":"info" ,"ct":"f5-api-service","ts":"2022-12-29 09:48:51.765624 UTC","log":{"time":"2022-12-29T09:48:51Z","log":"HEALTHY"}})
Attributes:
     -> log: Map({"log":"HEALTHY","time":"2022-12-29T09:48:51Z"})
     -> scid: Str(service-96km4711110)
     -> log.file.name: Str(t10.log)
     -> Severity: Str(info)
     -> ct: Str(f5-api-service)
     -> ts: Str(2022-12-29 09:48:51.765624 UTC)
Trace ID: 
Span ID: 
Flags: 0
	{"kind": "exporter", "data_type": "logs", "name": "logging"}

Additional context

When checking the routing processor with the following configuration
otel-collector-config_processor.txt

The routing to the correct exporter worked

See the resource output:
outputLog.txt

The instructions how to build and include the routing connector was mentioned in #26495

github-actions · 2023-09-20T12:38:03Z

Pinging code owners:

connector/routing: @jpkrohling @mwear

See Adding Labels via Comments if you do not have permissions to add labels yourself.

jpkrohling · 2023-10-05T14:31:42Z

@mwear are you able to take a look at this one?

mwear · 2023-10-05T16:54:46Z

I'll take a look @jpkrohling.

mwear · 2023-10-16T23:22:15Z

Sorry for the delay @nadavleva. I had a chance to look into this today. The routing connector routes based on resource attributes. The problem is that you do not have a resource attribute named Severity, which is what you are routing on. The issue is the move operation you have on your filelog receiver. Try changing it to:

receivers:
  filelog:
    include: [ /opt/parser/*.log ]
    start_at: beginning
    operators:
      - type: json_parser
        parse_from: body
        parse_to: attributes
      - type: severity_parser
        parse_from: attributes.sev
        mapping:
          fatal: emergency
          info4: notice
      - type: move
        id: move5
        from: attributes.sev
        to: resource.Severity

Note that the Severity attribute is missing from the Resource in the output you provided (both from the logging exporter and the attached txt file). You are actually moving it to an attribute on the log itself (as can be seen in the output as well).

Your output should look similar to this:

2023-10-16T16:10:39.139-0700	info	ResourceLog #0
Resource SchemaURL:
Resource attributes:
     -> Severity: Str(info)
ScopeLogs #0
ScopeLogs SchemaURL:
InstrumentationScope
LogRecord #0
ObservedTimestamp: 2023-10-16 23:10:39.038651 +0000 UTC
Timestamp: 2023-10-16 23:10:38 +0000 UTC
SeverityText: info
SeverityNumber: Info(9)
Body: Str(2023-10-16 16:10:38 info Testing 123)
Attributes:
     -> log.file.name: Str(simple.log)
     -> msg: Str(Testing 123)
     -> time: Str(2023-10-16 16:10:38)
Trace ID:
Span ID:
Flags: 0
	{"kind": "exporter", "data_type": "logs", "name": "logging"}

Give this a try and report back.

nadavleva · 2023-10-18T09:04:33Z

Hi @mwear
I tried your suggestion and used the resource attributes, the routing works as expected.
It was unclear how the routing statement works for example, route() where attributes["Severity"]
The attributes in the statement refer to "Resource attributes" from the resource schema and not to the attributes in the "ScopeLogs SchemaURL"

Here is the schema of the Log Resource.

Resource SchemaURL:
Resource attributes:
-> Severity: Str(info)
ScopeLogs #0
ScopeLogs SchemaURL:
InstrumentationScope
LogRecord #0
ObservedTimestamp: 2023-10-18 08:27:37.582473362 +0000 UTC
Timestamp: 1970-01-01 00:00:00 +0000 UTC
SeverityText: info
SeverityNumber: Info(9)
Body: Str({"scid":"service-96km4711110", "sev":"info" ,"ct":"f5-api-service","ts":"2022-12-29 09:48:51.765624 UTC","log":{"time":"2022-12-29T09:48:51Z","log":"HEALTHY"}})
Attributes:
-> log.file.name: Str(t10.log)
-> log: Map({"log":"HEALTHY","time":"2022-12-29T09:48:51Z"})
-> scid: Str(service-96km4711110)
-> sev: Str(info)
-> ct: Str(f5-api-service)
-> ts: Str(2022-12-29 09:48:51.765624 UTC)
-> Severity: Str(info)
Trace ID:
Span ID:
Flags: 0
{"kind": "exporter", "data_type": "logs", "name": "logging"}

Thank you for your answer and guidance

nadavleva added bug Something isn't working needs triage New item requiring triage labels Sep 20, 2023

github-actions bot added the connector/routing label Sep 20, 2023

This was referenced Sep 26, 2023

Weekly Report: 2023-09-19 - 2023-09-26 kevinslin/opentelemetry-collector-contrib#27

Open

Weekly Report: 2023-09-26 - 2023-10-03 #27402

Closed

github-actions bot mentioned this issue Oct 3, 2023

Weekly Report: 2023-09-26 - 2023-10-03 kevinslin/opentelemetry-collector-contrib#28

Open

github-actions bot mentioned this issue Oct 10, 2023

Weekly Report: 2023-10-03 - 2023-10-10 #27574

Closed

Frapschen assigned mwear Oct 16, 2023

Frapschen removed the needs triage New item requiring triage label Oct 16, 2023

nadavleva closed this as completed Oct 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[connector/routing] Routing by Attribute value fails and route to default pipeline #27020

[connector/routing] Routing by Attribute value fails and route to default pipeline #27020

nadavleva commented Sep 20, 2023 •

edited

Loading

github-actions bot commented Sep 20, 2023

jpkrohling commented Oct 5, 2023

mwear commented Oct 5, 2023 •

edited

Loading

mwear commented Oct 16, 2023

nadavleva commented Oct 18, 2023

[connector/routing] Routing by Attribute value fails and route to default pipeline #27020

[connector/routing] Routing by Attribute value fails and route to default pipeline #27020

Comments

nadavleva commented Sep 20, 2023 • edited Loading

Component(s)

What happened?

Description

Steps to Reproduce

Expected Result

Actual Result

Collector version

Environment information

Environment

OpenTelemetry Collector configuration

Log output

Additional context

github-actions bot commented Sep 20, 2023

jpkrohling commented Oct 5, 2023

mwear commented Oct 5, 2023 • edited Loading

mwear commented Oct 16, 2023

nadavleva commented Oct 18, 2023

nadavleva commented Sep 20, 2023 •

edited

Loading

mwear commented Oct 5, 2023 •

edited

Loading