Netobserv 390: fix kafka transformer

[OlivierCazade]

Multiple small changes in this PR:

• Fix timer in kafka ingester which was causing the FLP transformer to stop processing flows under too much traffic
• Add batch limit in kafka ingester
• Fix concurrent map access that could cause some crash when loglevel was set to debug
• Expose Kafka commit interval and default to 500ms for performances

[mariomac]

I'd add unit tests or integration tests, if possible

[mariomac]

I'd add a test that actually verifies the current additions. Something like:

• Abstract current kafka listener behind an injectable interface to provide mocks/fakes.

Test 1:

• Set a very high batch read timeout (e.g. 1h)
• Send mocked kafka messages with a total size lower than batchMaxLength
• Verify that messages aren't forwarded by the output channel
• Send messages until the total size is higher than batchMaxLength
• Verify that messages are forwarded by the output channel

Test 2:

• Set a very high batchMaxlength
• set a reasonably low batch timeout: e.g. 100 ms
• Send few mocked kafka messages (size < batchMatLength), making sure they are received before these 100ms. If we can't, increase the timeout e.g. to 200ms or ideally, also allow injecting a ticker that can be triggered on demand.
• Verify that the kafka messages are sent after the timeout

[codecov-commenter]

Codecov Report

Merging #237 (30d60a2) into main (4b7265e) will increase coverage by 0.10%.
The diff coverage is 84.84%.

@@            Coverage Diff             @@
##             main     #237      +/-   ##
==========================================
+ Coverage   61.22%   61.32%   +0.10%     
==========================================
  Files          67       67              
  Lines        3863     3886      +23     
==========================================
+ Hits         2365     2383      +18     
- Misses       1348     1352       +4     
- Partials      150      151       +1     

Flag	Coverage Δ
unittests	61.32% <84.84%> (+0.10%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/pipeline/ingest/ingest_kafka.go	80.15% <84.84%> (-0.43%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4b7265e...30d60a2. Read the comment docs.

[KalmanMeth]

Abstract current kafka listener behind an injectable interface to provide mocks/fakes.
Test 1: Set a very high batch read timeout (e.g. 1h)
Test 2: Set a very high batchMaxlength

Such tests could be added to test/e2e/kafka/kafka_test.go or the existing tests could be adjusted to include these.

[OlivierCazade]

I added the suggested tests.