Add Bunyan stream for OpenTelemetry

[martinkuba]

Is your feature request related to a problem? Please describe

This is a feature request to build a package to export Bunyan logs to OpenTelemetry backends. This would likely be done as a custom Bunyan stream utilizing the OpenTelemetry Logs API.

Describe the solution you'd like to see

Users can configure Bunyan to export logs to OpenTelemetry backends.

Related to #1558

[trentm]

Here is a proof-of-concept for this, as a start. I have a number of open questions and TODOs, but I'd appreciate feedback. If/when this seems a reasonable direction, then I can work on a PR to this repo for this. @hectorhdzg You also mentioned you were interested in this.

https://github.com/trentm/otellogsplay

Usage looks like this:

// ...
var log = bunyan.createLogger({name: 'myapp'});
log.addStream({
  type: 'raw',
  stream: new OTelBunyanAppender()
})

And that "appender" implementation is here: https://github.com/trentm/otellogsplay/blob/main/lib/OTelBunyanAppender.js

Here is an example run of that script:

% node bunyan-otel.js
{"name":"myapp","hostname":"purple.local","pid":46450,"level":30,"msg":"hi","time":"2023-07-10T23:16:17.930Z","v":0}
{
  timestamp: 1689030977930000,
  traceId: undefined,
  spanId: undefined,
  traceFlags: undefined,
  severityText: 'info',
  severityNumber: 9,
  body: 'hi',
  attributes: { name: 'myapp', hostname: 'purple.local', pid: 46450 }
}
{"name":"myapp","hostname":"purple.local","pid":46450,"level":30,"msg":"in a span","time":"2023-07-10T23:16:17.941Z","v":0}
{
  timestamp: 1689030977941000,
  traceId: '2891fcee99788edfabfa3c344edb70f8',
  spanId: '81b5a5a8de16a34d',
  traceFlags: 1,
  severityText: 'info',
  severityNumber: 9,
  body: 'in a span',
  attributes: { name: 'myapp', hostname: 'purple.local', pid: 46450 }
}
{
  traceId: '2891fcee99788edfabfa3c344edb70f8',
  parentId: undefined,
  traceState: undefined,
  name: 'manual-span',
  id: '81b5a5a8de16a34d',
  kind: 0,
  timestamp: 1689030977940000,
  duration: 1946,
  attributes: {},
  status: { code: 0 },
  events: [],
  links: []
}

Does this look like a reasonable start?
If so, I have a number of open questions and TODOs in the README and in https://github.com/trentm/otellogsplay/blob/main/lib/OTelBunyanAppender.js I'm happy to discuss them here, in a PR, or on Slack.

[hectorhdzg]

@trentm I think this is a good start, it would be easier to provide feedback directly in a PR, I think it would be good to align as much as possible with what Instrumentations are doing, maybe create similar abstract class, use global LoggerProvider by default, expose setLoggerProviderMethod, enabled/disable funcitonality, getLogger receiving LoggerOptions, etc.

[trentm]

Thanks for taking a look.

it would be easier to provide feedback directly in a PR

Understood. One thing I'm confused/struggling with is where this should live. Would this be a package, say at opentelemetry-js-contrib/packages/[opentelemetry-]bunyan-log-appender? A comment on this weeks SIG call and your mention of Instrumentations makes me wonder if some think this should be an instrumentation -- i.e. if it would fit in with the existing opentelemetry-js-contrib/plugins/node/opentelemetry-instrumentation-bunyan.

My initial thought was the former -- that usage of this would be as follows, so a user could explicitly setup their logger to send to whatever OTel SDK has been setup:

const bunyan = require('bunyan');
const { BunyanAppender } = require('@opentelemetry/bunyan-log-appender'); // or whatever package name

const log = bunyan.createLogger({name: 'myapp'});
log.addStream({
  type: 'raw',
  stream: new BunyanAppender(/* ... */)
})

And then possibly the opentelemetry-instrumentation-bunyan instrumentation would change to support automatically calling .addStream(...) on created Bunyan loggers.

Thoughts?

[hectorhdzg]

Yeah I'm more inclined into having a single bunyan package that provides trace injection functionality and auto generation of Log telemetry through the appender in same place, I can see having two different packages causing a lot of confusion to users, @pichlermarc what do you think?

[martinkuba]

I think it makes sense to include trace context injection in this module, so users can install just one package and get all the benefits of OTel. With that said, I think the existing bunyan instrumentation will need to continue to exist for use cases where users don't want to send the logs to OTLP backends.

As a side note, I think the name of the package/class should be stream, not appender, to be consistent with Bunyan's terminology and other existing streams.

[dyladan]

There are 2 concerns with logs:

1. Enrichment with otel context
2. Capturing for OTLP export

I think it would be helpful if the logs SIG provided guidance around which of these concerns (or both) instrumentations should be handling. Should it be configurable?

[trentm]

(FYI: I haven't made progress on this because I was on vacation last week and am away this week.)

[trentm]

I have a PR now for this at #1713
It adds to the existing @opentelemetry/instrumentation-bunyan.