Limit types allowed for Log Record Body

[pmm-sumo]

What are you trying to achieve?

Currently, Body is of any type, which allows for strings (actually, scalars), byte array, list of any values or a map<string, any>. In the last case, this brings confusion when to use Attributes (which is of map<string, any>)

This was discussed in #1613 and #1727

One of the observations was that we anticipate either fully-structured or partially-structured logs and it's hard to anticipate a use case when Body coming from a logging library would contain a map or an array.

Perhaps the Body type should be reconsidered and limited to string and bytearray, which better fits the existing use-cases. This would also avoid the confusion when to use Attributes. The complex types could be still included as entries in Attributes.

[yurishkuro]

I would like to clarify my comment that you're referencing (#1727 (comment)). When I said "unstructured blob goes to Body" I meant it from the logging API point of view:

• logger.Info(a=b, c=d, x=y) -- structured, goes to Attributes
• logger.Info(obj) -- goes to Body

It does not mean that obj itself is unstructured - depending on the logger it might try to serialize it into a JSON string, or even in a nested map<string, any> representation. The main difference is from the end-user perspective, either they are calling out a structure or they don't in their interaction with the logging API. From the protocol side, both attributes and body can be any.

I think the underlying issue in these discussions is how to make the API to be prescriptive such that the logging can be expressed in truly vendor-agnostic way, because if there is no specific expectation of what the backend can do with data represented this or other way, then what's the difference how to log it. But I am not sure we're in a position to make such call from the spec/API level - some backends might be smart enough to pick apart a structure even in a Body blob, while others may not be smart enough to even index Attributes.

[pmm-sumo]

My goal with this issue is to validate if we need to have complex types in Body. And concluding that we do (and the issue should be closed) is a fair answer.

When looking at the example mappings - the cases where any is needed are Splunk HEC and Google Cloud Logging. Perhaps it's important to retain it to keep the original data structure. If not, maybe the cases for complex types could be moved to Attributes or serialised as JSON into Body?

The other way to look at it is if end-user will want to discern between Body attributes (when it is a map) and regular Attributes. Maybe the answer is we don't know and I believe in such case it's better to keep any

[SergeyKanzhelev]

I feel like keeping it make sense. Do you think the resolution for this issue might be some additional clarification on how it is used? Or it is clear enough already?

[pmm-sumo]

I feel like keeping it make sense. Do you think the resolution for this issue might be some additional clarification on how it is used? Or it is clear enough already?

@SergeyKanzhelev this was actually sparked by the attempt to clarify it: #1727 :)

[SergeyKanzhelev]

Sorry, missed that.

[SergeyKanzhelev]

I actually think that #1752 (comment) makes a lot of sense - it may be the best we can do from vendor-neutral data collection standpoint

[errordeveloper]

...maybe the cases for complex types could be moved to Attributes or serialised as JSON into Body?

I am not sure if JSON strings in Body are practically useable, it's quite likely that users would want to query fields inside the JSON string, but how would a backed know that these are to be index or not? Perhaps producer happens to use an encoding other then JSON, like a simple comma-separated k/v format or some binary encoding. Perhaps a well-know attribute will need to define to hint backends at what type of data is in the body...

[djaglowski]

I believe this issue should be closed now. #2096 clarifies that body can be structured.