[exporter/googlecloud] Exporter for Logs should follow Data Model

[schmikei]

Component(s)

exporter/googlecloud

Describe the issue you're reporting

The Main Issue

Since the Google Cloud Exporter is treating the Body of the message differently than other exporters, it makes receivers/users have to be cognizant of which exporter they are going to and cannot be universally applied to all telemetry pipelines.

The main issues that are strongly apparent is that:

• Logs from first-party applications will never populate the jsonPayload field and only will ever use textPayload
• First-party applications will often have structured attributes, but GCP's labels can only be strings, so mapping attributes to labels is a potential incompatibility.

Current Behavior of the Exporter

Currently the Google Cloud exporter maps Body to GCL textPayload or jsonPayload, and Attributes is mapped to labels. From a UX perspective, this is undesirable, because the behavior of first-party receivers is to keep the raw log on Body, while parsing to attributes. This often leads to textPayload being used, where a customer expects their parsed log parts to be mapped to a jsonPayload. This is especially undesireable when the parsed attributes contain nested fields (e.g. parsing a json payload which may be arbitrarily nested).

Data model

The current data model strongly recommend that the body of an OTel log be a String for first-party receivers. This is generally how the log receivers in contrib are built, with parsed info for a log line going to the attributes. For instance, this is the default behavior of the pkg/stanza based parsing operators. In the Google cloud exporter, these are currently mapped to labels.

However, when collecting and sending logs using other agents, e.g. the Ops Agent, these parsed attributes are normally mapped to jsonPayload.

Proposed solution

Ultimately the maintainers should propose a solution they feel good about, however here are some alternatives that we have considered.

Instead of mapping Attributes to labels, instead combine Body and Attributes together, and use the combined map for jsonPayload instead.

One more thing we can do to better fit the LogEntry model of Google may be to add Resource Attributes to the labels result.

Examples

Parsing a mongodb log

This example highlights that just for a regex parsed log entry, it by default maps parsed attributes to labels, for more information on the entry model please see here.

Config

receivers:
  filelog:
    include:
      [
        /var/log/mongo/log/mongodb.log,
      ]
    start_at: beginning
    operators:
      - id: legacy_parser
        type: regex_parser
        regex: '^(?P<timestamp>\S+)\s+(?P<severity>\w+)\s+(?P<component>[\w-]+)\s+\[(?P<context>\S+)\]\s+(?P<message>.*)$'

exporters:
  googlecloud:
    log:
      default_log_name: opentelemetry.io/collector-exported-log

service:
  pipelines:
    logs:
      receivers: [filelog]
      exporters: [googlecloud]

Log Entry

2019-02-06T09:22:27.781-0500 I STORAGE  [initandlisten] Taking 873 samples and assuming that each section of oplog contains approximately 205254 records totaling to 48812234 bytes

Output

{
  "textPayload": "2019-02-06T09:22:27.781-0500 I STORAGE  [initandlisten] Taking 873 samples and assuming that each section of oplog contains approximately 205254 records totaling to 48812234 bytes",
  "insertId": "1rjj1jif6wpuqy",
  "resource": {
    "type": "generic_node",
    "labels": {
      "namespace": "",
      "node_id": "",
      "project_id": "redacted",
      "location": "global"
    }
  },
  "timestamp": "2022-11-18T18:25:40.295106Z",
  "labels": {
    "log.file.name": "mongodb.log",
    "component": "STORAGE",
    "context": "initandlisten",
    "message": "Taking 873 samples and assuming that each section of oplog contains approximately 205254 records totaling to 48812234 bytes",
    "timestamp": "2019-02-06T09:22:27.781-0500",
    "severity": "I"
  },
  "logName": "projects/project-id/logs/opentelemetry.io%2Fcollector-exported-log",
  "receiveTimestamp": "2022-11-18T18:25:40.564892962Z"
}

Example Output of MongoDB Logs from Ops Agent

{
  "jsonPayload": {
    "message": "Error receiving request from client. Ending connection from remote",
    "attributes": {
      "remote": "127.0.0.1:42906",
      "connectionId": 158866,
      "error": {
        "errmsg": "The server is configured to only allow SSL connections",
        "code": 141,
        "codeName": "SSLHandshakeFailed"
      }
    },
    "component": "NETWORK",
    "severity": "I",
    "id": 22988,
    "context": "conn158866"
  },
  "resource": {
    "type": "gce_instance",
    "labels": {
      "zone": "us-central1-a",
      "instance_id": "7556913141340343826",
      "project_id": "otel-agent-dev"
    }
  },
  "timestamp": "2022-11-18T20:28:30.650535063Z",
  "severity": "INFO",
  "labels": {
    "logging.googleapis.com/instrumentation_source": "agent.googleapis.com/mongodb",
    "compute.googleapis.com/resource_name": "mongo-test"
  },
  "logName": "projects/project-id/logs/mongodb",
  "receiveTimestamp": "2022-11-18T20:28:31.439667912Z"
}

Even the official agent of Google parses these logs and puts them into jsonPayload.

Parsing an Elasticsearch audit log

Because attributes are mapped to labels, parsing bodies with nested fields can make results difficult to search in GCL, since all label values must be strings:

Config

receivers:
  filelog:
    include:
      [
        ./local/elasticsearch_audit.json,
      ]
    start_at: beginning
    operators:
      - type: json_parser

exporters:
  loki:
    endpoint: http://localhost:3100/loki/api/v1/push
  googlecloud:
    log:
      default_log_name: "elasticsearch"
service:
  pipelines:
    logs:
      receivers: [filelog]
      exporters: [googlecloud, loki]

Original Log

{"type":"audit", "timestamp":"2020-12-31T00:33:52,521+0200", "node.id":"9clhpgjJRR-iKzOw20xBNQ", "event.type":"security_config_change", "event.action":"create_apikey", "request.id":"9FteCmovTzWHVI-9Gpa_vQ", "create":{"apikey":{"name":"test-api-key-1","expiration":"10d","role_descriptors":[{"cluster":["monitor","manage_ilm"],"indices":[{"names":["index-a*"],"privileges":["read","maintenance"]},{"names":["in*","alias*"],"privileges":["read"],"field_security":{"grant":["field1*","@timestamp"],"except":["field11"]}}],"applications":[],"run_as":[]},{"cluster":["all"],"indices":[{"names":["index-b*"],"privileges":["all"]}],"applications":[],"run_as":[]}],"metadata":{"application":"my-application","environment":{"level": 1,"tags":["dev","staging"]}}}}}

The nested keys are turned into strings when placed on labels, making them impossible to match on properly:

Output

{
  "textPayload": "{\"type\":\"audit\", \"timestamp\":\"2020-12-31T00:33:52,521+0200\", \"node.id\":\"9clhpgjJRR-iKzOw20xBNQ\", \"event.type\":\"security_config_change\", \"event.action\":\"create_apikey\", \"request.id\":\"9FteCmovTzWHVI-9Gpa_vQ\", \"create\":{\"apikey\":{\"name\":\"test-api-key-1\",\"expiration\":\"10d\",\"role_descriptors\":[{\"cluster\":[\"monitor\",\"manage_ilm\"],\"indices\":[{\"names\":[\"index-a*\"],\"privileges\":[\"read\",\"maintenance\"]},{\"names\":[\"in*\",\"alias*\"],\"privileges\":[\"read\"],\"field_security\":{\"grant\":[\"field1*\",\"@timestamp\"],\"except\":[\"field11\"]}}],\"applications\":[],\"run_as\":[]},{\"cluster\":[\"all\"],\"indices\":[{\"names\":[\"index-b*\"],\"privileges\":[\"all\"]}],\"applications\":[],\"run_as\":[]}],\"metadata\":{\"application\":\"my-application\",\"environment\":{\"level\": 1,\"tags\":[\"dev\",\"staging\"]}}}}}",
  "insertId": "45usidf8ugrc7",
  "resource": {
    "type": "generic_node",
    "labels": {
      "node_id": "",
      "namespace": "",
      "location": "global",
      "project_id": "redacted"
    }
  },
  "timestamp": "2022-11-18T19:35:16.324903Z",
  "labels": {
    "event.action": "create_apikey",
    "create": "{\"apikey\":{\"expiration\":\"10d\",\"metadata\":{\"application\":\"my-application\",\"environment\":{\"level\":1,\"tags\":[\"dev\",\"staging\"]}},\"name\":\"test-api-key-1\",\"role_descriptors\":[{\"applications\":[],\"cluster\":[\"monitor\",\"manage_ilm\"],\"indices\":[{\"names\":[\"index-a*\"],\"privileges\":[\"read\",\"maintenance\"]},{\"field_security\":{\"except\":[\"field11\"],\"grant\":[\"field1*\",\"@timestamp\"]},\"names\":[\"in*\",\"alias*\"],\"privileges\":[\"read\"]}],\"run_as\":[]},{\"applications\":[],\"cluster\":[\"all\"],\"indices\":[{\"names\":[\"index-b*\"],\"privileges\":[\"all\"]}],\"run_as\":[]}]}}",
    "log.file.name": "elasticsearch_audit.json",
    "timestamp": "2020-12-31T00:33:52,521+0200",
    "event.type": "security_config_change",
    "request.id": "9FteCmovTzWHVI-9Gpa_vQ",
    "type": "audit",
    "node.id": "9clhpgjJRR-iKzOw20xBNQ"
  },
  "logName": "projects/project-id/logs/elasticsearch",
  "receiveTimestamp": "2022-11-18T19:35:16.691162178Z"
}

The Loki exporter, for example, preserves the nested keys, making them matchable:

Loki

{
    "body": "{\"type\":\"audit\", \"timestamp\":\"2020-12-31T00:33:52,521+0200\", \"node.id\":\"9clhpgjJRR-iKzOw20xBNQ\", \"event.type\":\"security_config_change\", \"event.action\":\"create_apikey\", \"request.id\":\"9FteCmovTzWHVI-9Gpa_vQ\", \"create\":{\"apikey\":{\"name\":\"test-api-key-1\",\"expiration\":\"10d\",\"role_descriptors\":[{\"cluster\":[\"monitor\",\"manage_ilm\"],\"indices\":[{\"names\":[\"index-a*\"],\"privileges\":[\"read\",\"maintenance\"]},{\"names\":[\"in*\",\"alias*\"],\"privileges\":[\"read\"],\"field_security\":{\"grant\":[\"field1*\",\"@timestamp\"],\"except\":[\"field11\"]}}],\"applications\":[],\"run_as\":[]},{\"cluster\":[\"all\"],\"indices\":[{\"names\":[\"index-b*\"],\"privileges\":[\"all\"]}],\"applications\":[],\"run_as\":[]}],\"metadata\":{\"application\":\"my-application\",\"environment\":{\"level\": 1,\"tags\":[\"dev\",\"staging\"]}}}}}",
    "attributes": {
        "create": {
            "apikey": {
                "expiration": "10d",
                "metadata": {
                    "application": "my-application",
                    "environment": {
                        "level": 1,
                        "tags": [
                            "dev",
                            "staging"
                        ]
                    }
                },
                "name": "test-api-key-1",
                "role_descriptors": [
                    {
                        "applications": [],
                        "cluster": [
                            "monitor",
                            "manage_ilm"
                        ],
                        "indices": [
                            {
                                "names": [
                                    "index-a*"
                                ],
                                "privileges": [
                                    "read",
                                    "maintenance"
                                ]
                            },
                            {
                                "field_security": {
                                    "except": [
                                        "field11"
                                    ],
                                    "grant": [
                                        "field1*",
                                        "@timestamp"
                                    ]
                                },
                                "names": [
                                    "in*",
                                    "alias*"
                                ],
                                "privileges": [
                                    "read"
                                ]
                            }
                        ],
                        "run_as": []
                    },
                    {
                        "applications": [],
                        "cluster": [
                            "all"
                        ],
                        "indices": [
                            {
                                "names": [
                                    "index-b*"
                                ],
                                "privileges": [
                                    "all"
                                ]
                            }
                        ],
                        "run_as": []
                    }
                ]
            }
        },
        "event.action": "create_apikey",
        "event.type": "security_config_change",
        "log.file.name": "elasticsearch_audit.json",
        "node.id": "9clhpgjJRR-iKzOw20xBNQ",
        "request.id": "9FteCmovTzWHVI-9Gpa_vQ",
        "timestamp": "2020-12-31T00:33:52,521+0200",
        "type": "audit"
    }
}

[github-actions]

Pinging code owners:

• exporter/googlecloud: @aabmass @dashpole @jsuereth @punya @tbarker25 @damemi

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[damemi]

Thanks for the detailed issue description @schmikei!

I think the main friction here is that the log receivers/transform processors parse to attributes by default instead of body. This can be changed with parse_to: body, which produces a JSON payload like you expect.

That was an intentional decision (open-telemetry/opentelemetry-log-collection#431) in order to make the operators non-destructive to the original log. Along those same lines, we wanted the exporter to be as non-opinionated as possible, leaning on the availability of other processors to allow users to modify entries as they need to.

Could you clarify how this doesn't follow the data model? It wasn't clear to me from the issue description, but based on the logging data model spec, attributes should be matched to GCP labels. Based on that, I think this feature would actually be a change from the current spec.

One more thing we can do to better fit the LogEntry model of Google may be to add Resource Attributes to the labels result.

This was indeed something we overlooked, and fixed in GoogleCloudPlatform/opentelemetry-operations-go#531, which will be in the next release :)

[djaglowski]

I think the main friction here is that the log receivers/transform processors parse to attributes by default instead of body. This can be changed with parse_to: body, which produces a JSON payload like you expect.

 That was an intentional decision (open-telemetry/opentelemetry-log-collection#431) in order to make the operators non-destructive to the original log.

It's worth noting that parsing to attributes is strongly encouraged by the data model itself, so there is more to it than just preserving the original body. This is actually noted in the linked issue: The data model strongly encourages that structured data belong in attributes.

The incongruity seems to be that, while the json_payload is intended to represent structured data, the data model calls for the body to be unstructured. Therefore, applications and configurations that follow the data model's suggested usage of body will never make use of json_payload, even though they may be representing structured logs.

[damemi]

Talked about this offline with @braydonk and other gcp folks, we'd like to bring this up at the next SIG meetings (Logging to ask about the data model semantics and Collector to clarify how a change like this would affect the stability/compatibility of our exporter).

Personally I think adding a feature flag to our exporter like attributes_to_body: true would be good. That keeps our exporter in-line with the data model by default and marks the deviation as a feature, so the stability contract is softened (imo).

But, if this is something we just want to address in the data model (ie, no one actually cares about preserving the original string log), then we should just make it the default behavior and avoid the extra hoop of a feature flag.

[damemi]

(linking) spec update for structuredbody: open-telemetry/opentelemetry-specification#3014

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/googlecloud: @aabmass @dashpole @jsuereth @punya @tbarker25 @damemi

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[psoladoye]

@schmikei Thanks for the detailed explanation on this issue. I am currently trying to navigate how to align logs coming from the collector with Google Cloud Logging as well. What's the resolution on this?

[djaglowski]

@psoladoye, open-telemetry/opentelemetry-specification#3023 came out the discussions around this, which if I remember correctly should have resolved this issue. Perhaps @damemi can check me on that and if it's agreed we could close this.

[psoladoye]

@djaglowski Got it. Thanks.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/googlecloud: @aabmass @dashpole @jsuereth @punya @damemi @psx95

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[braydonk]

My understanding of where this issue left off:

Google Cloud Exporter maps Attributes to logEntry.labels and Body to logEntry.jsonPayload if structured or logEntry.textPayload if unstructured. When the logging processors parse a structured log, they parse the new fields into Attributes and put the message into Body as a string. This was because of a previous tricky wording in the Log spec that made it seem like it was not intended to put structured data into Body.

This was hashed out in a spec meeting where it was clarified that if the source is a structured log already, i.e. a third party structured log, then parsing structured data into the body is perfectly fine. The original intent of "messages in Body should only be a string" was originally intended specifically for first-party usage.

This was addressed in the spec in open-telemetry/opentelemetry-specification#3023

The result of this is that how the Google Cloud Exporter handles this actually makes sense based on the intention of the spec, and that data coming from logging sources should parse structured data into Body in scenarios where it makes sense.

This is where I lost the thread of the issue; I don't remember what the plan was on the logging receiver/processor side. However, since this issue is targeted at the Google Cloud Exporter, and we have determined the exporter behaviour here makes sense with the spec, I am tempted to say this issue can be closed.