-
Notifications
You must be signed in to change notification settings - Fork 416
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
kamon.metric.Timer warns about negative values due to NTP time sync #3480
Comments
Hi @deemru, thanks for reaching out. The root cause of an issue at hand is not actually the metrics subsystem, it's NTP. Note that the value which did not fit in the range is negative: it's because the time has been adjusted during block application. This is most likely due to the fact that the Unfortunately, as stated in kamon-io/Kamon#585, Kamon is built in such way that it's not supposed to be turned off. Metrics collection (even in hot code paths) does not have measurable impact on the node's overall performance (please share your benchmark data if your experience differs from what we've measured). What may impact performance or resource consumption are reporters which deliver aggregated metrics to storage backends like InfluxDB. Those are disabled by default. If you prefer not to see metrics-related messages in your logs, you can add the following line to your custom <logger name="kamon" level="OFF"/> |
Can you please explain to me how this code which should do many many things to complete is better than if you just add before it simple condition like Waves/node/src/main/scala/com/wavesplatform/state/appender/MicroblockAppender.scala Line 28 in 7f83b02
|
As I've said, «doing many things to complete» does not take that long, so there's no point in optimizing this piece, because these optimizations will not yield measurable block application speedup. |
For me your approach to resolve this issue is look like:
|
I'm sorry, I can't see how enabling/disabling miner or REST API is related to the original issue. If you are concerned with API or miner performance impact on import speed or ordinary block application, please consider opening another issue. We will consider tweaking |
Metrics In our world of metrics, telemetry etc. An user should be sure he can opt-out of these things. |
No reporters are enabled by default, so nothing is shared with third party, unless explicitly configured. I believe this fits the definition of opt-in. I'm locking this issue, because the root cause has been identified and action plan has been determined. Please feel free to open a separate issue if you feel so inclined. |
We still (on the newest node 1.3.4) get errors like these:
WARN [ecution-context-global-XX] kamon.metric.Timer - Failed to record value [-70000000] on [span.processing-time,{operation=block-appender,error=false}] because the value is outside of the configured range. The recorded value was adjusted to the highest trackable value [3600000000000]. You might need to change your dynamic range configuration for this metric
This is not just annoying to see (was reported many times for years) but this also means the code executes and spends CPU cycles on non active functionality:
Because metrics calls (as far as i understand) are unconditional (for example just search for
measureSuccessful
)Please, for god sake, make these metrics completely turned off by
metrics.enable = no
and i mean off = check config once and don`t waste CPU cycles anymore on that.The text was updated successfully, but these errors were encountered: