Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow to start FLP directly from the flow logs producer #538

Merged
merged 5 commits into from
Nov 23, 2023

Conversation

jotak
Copy link
Member

@jotak jotak commented Nov 16, 2023

Description

  • New "InProcess" ingest stage
  • New entry point to start the whole FLP in-process: pipeline.StartFLPInProcess
  • Add some tests
  • Also moved the code to start prometheus into a new function/package to avoid repetitions

The idea behind is to reduce some of the overhead when the agent producing flows, and the FLP workload are running on the same host: in that case, the network connection between them could be avoided, saving resources.

Dependencies

n/a

Checklist

If you are not familiar with our processes or don't know what to answer in the list below, let us know in a comment: the maintainers will take care of that.

  • Will this change affect NetObserv / Network Observability operator? If not, you can ignore the rest of this checklist.
  • Is this PR backed with a JIRA ticket? If so, make sure it is written as a title prefix (in general, PRs affecting the NetObserv/Network Observability product should be backed with a JIRA ticket - especially if they bring user facing changes).
  • Does this PR require product documentation?
    • If so, make sure the JIRA epic is labelled with "documentation" and provides a description relevant for doc writers, such as use cases or scenarios. Any required step to activate or configure the feature should be documented there, such as new CRD knobs.
  • Does this PR require a product release notes entry?
    • If so, fill in "Release Note Text" in the JIRA.
  • Is there anything else the QE team should know before testing? E.g: configuration changes, environment setup, etc.
    • If so, make sure it is described in the JIRA ticket.
  • QE requirements (check 1 from the list):
    • Standard QE validation, with pre-merge tests unless stated otherwise.
    • Regression tests only (e.g. refactoring with no user-facing change).
    • No QE (e.g. trivial change with high reviewer's confidence, or per agreement with the QE team).

@jotak jotak added no-qe This PR doesn't necessitate QE approval no-doc This PR doesn't require documentation change on the NetObserv operator labels Nov 16, 2023
- New "InProcess" ingest stage
- New entry point to start the whole FLP in-process: pipeline.StartFLPInProcess
- Add some tests
Copy link

codecov bot commented Nov 16, 2023

Codecov Report

Attention: 32 lines in your changes are missing coverage. Please review.

Comparison is base (7c78d63) 66.84% compared to head (0ca3b67) 67.42%.

Files Patch % Lines
pkg/prometheus/prom_server.go 58.49% 15 Missing and 7 partials ⚠️
cmd/flowlogs-pipeline/main.go 0.00% 4 Missing ⚠️
pkg/pipeline/inprocess.go 80.00% 2 Missing and 1 partial ⚠️
pkg/pipeline/encode/encode_prom.go 0.00% 1 Missing ⚠️
pkg/pipeline/ingest/ingest_inprocess.go 90.90% 1 Missing ⚠️
pkg/pipeline/pipeline.go 90.90% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #538      +/-   ##
==========================================
+ Coverage   66.84%   67.42%   +0.57%     
==========================================
  Files          95       97       +2     
  Lines        6802     6857      +55     
==========================================
+ Hits         4547     4623      +76     
+ Misses       1988     1959      -29     
- Partials      267      275       +8     
Flag Coverage Δ
unittests 67.42% <73.98%> (+0.57%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Comment on lines 184 to 185
prometheus.SetGlobalMetricsSettings(&cfg.MetricsSettings)
promServer := prometheus.StartServerAsync(&cfg.MetricsSettings.PromConnectionInfo, nil)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about making this optionnal ?
If we are looking for lightweight solution we may skip prometheus since we have other export capabilities now

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not only for prometheus pipeline output but also for the "operational metrics" ie. related to FLP health .. which doesn't mean it cannot be made optional, but I would say the default should still to have these metrics

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can add a field enableOperational in MetricsSettings.
If it's set false, this "global" server would not be started, and it's still possible to start per-promencode stage servers.
However if it's disabled and the user didn't configure per-promencode servers (which is optional), then they wouldn't get any metric ... so it creates more configuration skews....

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe enableGlobalServer would be a better name for the knob as it carries more explicitly that's it's neededif you don't configure per-stage server.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks !

@jotak jotak added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Nov 20, 2023
Copy link

New image:
quay.io/netobserv/flowlogs-pipeline:8fad274

It will expire after two weeks.

To deploy this build, run from the operator repo, assuming the operator is running:

USER=netobserv VERSION=8fad274 make set-flp-image

@github-actions github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Nov 20, 2023
@jotak
Copy link
Member Author

jotak commented Nov 23, 2023

/approve

Copy link

openshift-ci bot commented Nov 23, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jotak

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-bot openshift-merge-bot bot merged commit f3b03fa into netobserv:main Nov 23, 2023
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved lgtm no-doc This PR doesn't require documentation change on the NetObserv operator no-qe This PR doesn't necessitate QE approval
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants