feat: record prom metrics and ui consumes them #36
Conversation
.gitignore
Outdated
| @@ -1,3 +1,4 @@ | |||
| hydra-booster | |||
| hydra-belly | |||
| coverage.txt | |||
| data | |||
There was a problem hiding this comment.
can prometheus write to prometheus-data?
There was a problem hiding this comment.
I did not see an option for this I could put in promconfig.yaml but there is a command line option I could add in the docs (I wanted this also 😂).
| fmt.Fprintf(os.Stderr, "Running %d DHT Sybils:\n", options.NSybils) | ||
|
|
||
| var hyperLock sync.Mutex | ||
| hyperlog := hyperloglog.New() |
There was a problem hiding this comment.
Idea: Instead of storing and calculating unique peers per hydra, we should just log them to a file and use a separate process to tail the logs and calculate it. We want our hydra to do as little extra work as possible since we're going to be running so many sybils on each machine.
@olizilla is using https://github.com/google/mtail to do this in combination with the peerlog plugin to get unique peers seen by the gateways.
If we did this, we could then monitor unique peer IDs across multiple hydras, which may be a reassurring compliment to the unique peers seen by the gateways.
cc @momack2
There was a problem hiding this comment.
clarification, the peerlog plugin logs unique peerIDs as they connect and disconnect, and we use ELK / elastic search to report over unique peerIDs seen across a given time period. If hydra-bootster can log PeerIDs in the same structure as produced by the peerlog plugin, we can send those logs to ELK too, and add them to that report.
mtail is an alternative option, for situations where there is a metric you want to scrape with prometheus that can only be derived from logs, but we dont need both.
There was a problem hiding this comment.
by agreggating over all the logs for all the infra, we can apply the uniquness constraint across all services, which is neat.
|
|
supersedes #30