Create new transformer into OpenTelemetry semantics

[eranra]

Focus on Flow-Logs / Connection-Logs first
Metrics as a second phase

[eranra]

ref: https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/k8s/#pod

[jotak]

I'm not sure if it needs to be an exporter, or a transformer. Could it be just a transformer, that does the semantic fields naming and adaptation, and then it can be plugged with prometheus/loki without any change on the exporter side?

[jotak]

See also the conventions for network attributes: https://opentelemetry.io/docs/reference/specification/trace/semantic_conventions/span-general/#network-transport-attributes
But this is where the spec is limited and not a super good fit for us: there's either k8s semantic conventions for the observation point, or network semantic conventions for in/out communications, but these domains cannot be composed; e.g. there's no semantic convention for a "destination pod" as far as I know (something like peer.k8s.pod.name doesn't exist in spec)

[bdarfler]

Would this also include exporting in OpenTelemetry Line Protocol? It seems semantics and protocol support would be best served together.

[eranra]

Would this also include exporting in OpenTelemetry Line Protocol? It seems semantics and protocol support would be best served together.

Hi @bdarfler. I agree on the need for both semantics and protocol to be supported --- this is still not defined to the end, and we are soon going to start working on that ( @KalmanMeth ) --- can you share more details/motivation for the protocol? Consumption by which component?

[bdarfler]

I could see users being interested in sending this data to the OpenTelemetry Collector and then sending it to their vendor or backend of choice.

The pipeline architecture of flowlogs and the OTel Collector is quite similar so there may also be some benefit for flowlogs to interoperate with the OTel Collector and leverage its processing and transformation capabilities rather than reimplement all that within flowlogs.

I can see an argument for keeping high-value protocol support like loki and prometheus in flowlogs but for the longer tail (like paid vendors, etc.) it might be nice to offload that support to the existing support in the OTel Collector

[eranra]

@bdarfler ok, got it; THX for the explanation--- it does make sense; although the Network Observability pipeline includes various "network specific" and "kube specific" capabilities that are required + it also generates metrics from the logs --- I did not follow up with OTel capabilities in this space. Still, I would not be surprised if there is some overlap.

[KalmanMeth]

Issue is addresed by #531 and #577.