Write rates of endpoints still high on 3.6.1

[smarterclayton]

API server write rates from 3.6.1 on a heavily loaded cluster.

{resource="endpoints",verb="PUT"} | 35.0125
{resource="pods",verb="PUT"} | 19.45416666666667
{resource="pods",verb="DELETE"} | 16.987499999999997
{resource="events",verb="POST"} | 16.841666666666665
{resource="pods",verb="POST"} | 16.104166666666664
{resource="nodes",verb="PATCH"} | 15.579166666666667
{resource="localsubjectaccessreviews",verb="POST"} | 8.433333333333334
{resource="builds",verb="PATCH"} | 8.108333333333333
{resource="daemonsets",verb="PUT"} | 6.304166666666666
{resource="replicasets",verb="POST"} | 4.279166666666667
{resource="subjectaccessreviews",verb="POST"} | 1.7291666666666667
{resource="tokenreviews",verb="POST"} | 0.9333333333333333
{resource="replicationcontrollers",verb="PUT"} | 0.4666666666666667
{resource="deploymentconfigs",verb="PUT"} | 0.3541666666666667
{resource="deploymentconfigs",verb="POST"} | 0.2583333333333333

@derekwaynecarr @openshift/sig-platform-management

[liggitt]

do we have large numbers of endpoints with no subsets? if so, kubernetes/kubernetes#50583 might be relevant

[smarterclayton]

Hrm, doubt it but maybe.

[smarterclayton]

I don't like seeing 16 pod deletes a second. That's probably bad.

[smarterclayton]

@openshift/sig-master

[smarterclayton]

Here's write volume to etcd before and after the 3.6.1 upgrade.

[smarterclayton]

The jump was Sept 26th 2:15 gmt. May be unrelated to update?

[liggitt]

what is the server version?

[smarterclayton]

this was 3.6.1 official

[liggitt]

the endpoint update fix merged to ose on 2017-09-15 08:22, and is only in these tags:

v3.6.173.0.35-1
v3.6.173.0.36-1
v3.6.173.0.37-1
v3.6.173.0.38-1
v3.6.173.0.39-1
v3.6.173.0.40-1
v3.6.173.0.41-1
v3.6.173.0.42-1
v3.6.173.0.43-1
v3.6.173.0.44-1
v3.6.173.0.45-1
v3.6.173.0.46-1
v3.6.173.0.47-1
v3.6.173.0.48-1
v3.6.173.0.49-1

[liggitt]

what sha or tag is 3.6.1 official?

[liggitt]

looks like 3.6.1 is 3.6.173.0.21, which does not include the endpoints fixes yet

[smarterclayton]

I lied, us-east-2 is at 3.6.173.0.37 {resource="endpoints",verb="PUT"} 33.325 {resource="pods",verb="PUT"} 17.179166666666667 {resource="nodes",verb="PATCH"} 15.575 {resource="pods",verb="DELETE"} 14.454166666666664 {resource="events",verb="POST"} 12.762500000000001 {resource="pods",verb="POST"} 12.695833333333331 {resource="builds",verb="PATCH"} 9.983333333333333 {resource="localsubjectaccessreviews",verb="POST"} 9.920833333333334 {resource="daemonsets",verb="PUT"} 4.775 {resource="replicasets",verb="POST"} 3.1 {resource="subjectaccessreviews",verb="POST"} 1.8 {resource="tokenreviews",verb="POST"} 0.9708333333333332 {resource="replicationcontrollers",verb="PUT"} 0.20000000000000004 {resource="deploymentconfigs",verb="PUT"} 0.17083333333333336 {resource="deploymentconfigs",verb="POST"} 0.09583333333333334 {resource="events",verb="PATCH"} 0.06666666666666667 {resource="bindings",verb="POST"} 0.029166666666666667 {resource="imagestreamimports",verb="POST"} 0.025

…

On Thu, Oct 12, 2017 at 12:49 PM, Jordan Liggitt ***@***.***> wrote: looks like 3.6.1 is 3.6.173.0.21, which does not include the endpoints fixes yet — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#16583 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_pxlqUe1iUH1oOuzsOxREJiWo47DYks5srkMKgaJpZM4Pl3ia> .

[smarterclayton]

Was one single daemonset https://bugzilla.redhat.com/show_bug.cgi?id=1501514. Moving to bug

[smarterclayton]

After the daemonset is fixed we're still doing 30 writes / s from endpoints and pod traffic is <1 w/s

[liggitt]

what environment and version are you seeing this on?

[liggitt]

need at least 3.6.173.0.48 for the controller and resourceVersion no-op fixes @joelsmith picked

[liggitt]

we're seeing high levels of endpoint write API calls, but low volumes of updates when watching endpoints, then it is likely that the writes are no-op writes, and kubernetes/kubernetes#50583 is a candidate fix