Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[kafka-producer] Support setting max message size #5263

Merged
merged 2 commits into from
Mar 9, 2024
Merged

[kafka-producer] Support setting max message size #5263

merged 2 commits into from
Mar 9, 2024

Conversation

sappusaketh
Copy link
Contributor

@sappusaketh sappusaketh commented Mar 9, 2024

Which problem is this PR solving?

Description of the changes

  • Adds support to set MaxMessageBytes config on producer
  • Also setting a default which matches with sarama default unclear if we need to set that default explicitly from jaeger producer initialization

How was this change tested?

  • Tested on our development jaeger cluster. After seeing below error we made following changes and built a new Image and verified that my changes fixed the issue
{"level":"error","ts":1550003610.8029132,"caller":"kafka/writer.go:59","msg":"kafka server: Message was too large, server rejected it to avoid allocation error.","stacktrace":"[github.com/jaegertracing/jaeger/plugin/storage/kafka.NewSpanWriter.func2](http://github.com/jaegertracing/jaeger/plugin/storage/kafka.NewSpanWriter.func2)\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/plugin/storage/kafka/writer.go:59"}

Checklist

Signed-off-by: Saketh kappala <43525626+sappusaketh@users.noreply.github.com>
@sappusaketh sappusaketh requested a review from a team as a code owner March 9, 2024 03:57
Co-authored-by: Yuri Shkuro <yurishkuro@users.noreply.github.com>
Signed-off-by: Saketh <43525626+sappusaketh@users.noreply.github.com>
Copy link
Member

@yurishkuro yurishkuro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks

Copy link

codecov bot commented Mar 9, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.09%. Comparing base (083806f) to head (9648fb4).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #5263   +/-   ##
=======================================
  Coverage   95.09%   95.09%           
=======================================
  Files         339      339           
  Lines       16503    16510    +7     
=======================================
+ Hits        15693    15700    +7     
  Misses        621      621           
  Partials      189      189           
Flag Coverage Δ
cassandra-3.x 26.53% <0.00%> (-0.04%) ⬇️
cassandra-4.x 26.53% <0.00%> (-0.04%) ⬇️
elasticsearch-5.x 21.75% <0.00%> (-0.05%) ⬇️
elasticsearch-6.x 21.77% <0.00%> (-0.01%) ⬇️
elasticsearch-7.x 21.84% <0.00%> (-0.05%) ⬇️
elasticsearch-8.x 21.94% <0.00%> (-0.03%) ⬇️
grpc-badger 19.23% <0.00%> (-0.02%) ⬇️
kafka 14.66% <100.00%> (+0.10%) ⬆️
opensearch-1.x 21.85% <0.00%> (-0.03%) ⬇️
opensearch-2.x 21.85% <0.00%> (-0.01%) ⬇️
unittests 92.71% <85.71%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@yurishkuro yurishkuro changed the title Add support for setting max message bytes config [kafka-producer] Support setting max message size Mar 9, 2024
@yurishkuro yurishkuro merged commit bf84fc6 into jaegertracing:main Mar 9, 2024
35 of 36 checks passed
@sappusaketh sappusaketh deleted the producer-max-message-bytes branch March 11, 2024 13:35
yurishkuro pushed a commit that referenced this pull request Mar 19, 2024
## Which problem is this PR solving?
- PR #5263 added support for setting max message size a producer can
send that solved the issue on collector end now we are seeing our
consumer getting stuck consuming messages larger that 1MB(default fetch
max message bytes) so adding an option to set this value via flags

## Description of the changes
- adding an option to set fetch max message bytes value via flags

## How was this change tested?
- Tested against our dev jaeger where we have spans larger than 1 MB

## Checklist
- [x] I have read
https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md
- [x] I have signed all commits
- [x] I have added unit tests for the new functionality
- [x] I have run lint and test steps successfully
  - for `jaeger`: `make lint test`
  - for `jaeger-ui`: `yarn lint` and `yarn test`

Signed-off-by: Saketh kappala <43525626+sappusaketh@users.noreply.github.com>
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants