HNC: export from OpenCensus to Prometheus

[adrianludwin]

cc @rjbez17 @yiqigao217

As of #410, HNC is now able to exporting metrics to Stackdriver. It doesn't yet support Prometheus because:

1. None of us HNC developers were able to install and configure Prometheus in a reasonable amount of time;
2. We couldn't figure out how to get OpenCensus' Prometheus endpoint to play nicely with controller-runtime's Prometheus endpoint. Maybe we can expose them on different ports?

For anyone with Prometheus experience, it's probably fairly easy to fix this problem but we don't currently have the expertise to do it ourself.

/good-first-issue

[k8s-ci-robot]

@adrianludwin:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

cc @rjbez17 @yiqigao217

As of #410, HNC is now able to exporting metrics to Stackdriver. It doesn't yet support Prometheus because:

1. None of us HNC developers were able to install and configure Prometheus in a reasonable amount of time;
2. We couldn't figure out how to get OpenCensus' Prometheus endpoint to play nicely with controller-runtime's Prometheus endpoint. Maybe we can expose them on different ports?

For anyone with Prometheus experience, it's probably fairly easy to fix this problem but we don't currently have the expertise to do it ourself.

/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[rjbez17]

Sounds like I can actually be useful! I'll try to take a wack at this next week.

[yeya24]

@rjbez17 Hi, I think I can work on this issue this week. Is it okay?

[rjbez17]

I’ll never complain about someone doing work for me faster than me! Go for it

[yeya24]

I am checking this issue now. But I am wondering what metrics do we have currently? If we don't have any metrics except the metrics from controller-runtime, why we need to add Prometheus? @adrianludwin @rjbez17

[adrianludwin]

Hi Ben, have you worked with metrics and Prometheus before? We have metrics exported from controller-runtime on a Prometheus endpoint, and also from OpenCensus via Stackdriver (see https://github.com/kubernetes-sigs/multi-tenancy/blob/master/incubator/hnc/pkg/stats/metrics.go). We'd like to also export from OpenCensus to Prometheus, but without disrupting the endpoint from controller-runtime. Does that all make sense? If not, it's probably not a good starter project on HNC since this area is quite complicated if you don't already know all these monitoring systems. Thanks!

…

On Mon, Feb 17, 2020 at 4:41 PM Ben Ye ***@***.***> wrote: I am checking this issue now. But I am wondering what metrics do we have currently? If we don't have any metrics except the metrics from controller-runtime, why we need to add Prometheus? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#416>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE43PZDQFABVDFIKCLRIFFDRDMABRANCNFSM4KT6OHFQ> .

[yeya24]

@adrianludwin Hi, so you want to add the metrics from Stackdriver to the /metrics endpoint exposed by controller-runtime?

[adrianludwin]

If possible, yes. Otherwise, we'd like to expose a *second* /metrics endpoint (e.g. on another port) and have instructions for how to configure Prometheus to use both.

…

On Tue, Feb 18, 2020 at 9:07 AM Ben Ye ***@***.***> wrote: @adrianludwin <https://github.com/adrianludwin> Hi, so you want to add the metrics from Stackdriver to the /metrics endpoint exposed by controller-runtime? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#416>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE43PZCD5ZXDMNFWXOSYXGTRDPTS3ANCNFSM4KT6OHFQ> .

[yeya24]

Yes, I have implemented this to use the same /metrics. But I am not familiar with StackDriver so I am not sure whether it works or not. I don't have an env for testing

[adrianludwin]

Cool - maybe throw up a PR and we can see if StackDriver is still working?

…

On Tue, Feb 18, 2020 at 9:28 AM Ben Ye ***@***.***> wrote: Yes, I have implemented this to use the same /metrics. But I am not familiar with StackDriver so I am not sure whether it works or not. I don't have an env for testing — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#416>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE43PZGA7RMOMTL3IFNM5W3RDPV7RANCNFSM4KT6OHFQ> .

[yeya24]

PR opened, check #432

[adrianludwin]

Thanks! I'll leave further comments there.

…

On Tue, Feb 18, 2020 at 9:34 AM Ben Ye ***@***.***> wrote: PR opened, check #432 <#432> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#416>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE43PZHJJ3WHDJP5DDJCYL3RDPWXLANCNFSM4KT6OHFQ> .