[area/questions] Make default snapshot-count 10k

[chaochn47]

I am wondering if etcd should make snapshot-count to be 10k because it is recommended in kubernetes to set this as 10k. kubernetes/kubernetes#100475 (comment)

I cannot explain why etcd uses higher default value 100k and kubeadm snapshot-count default value is 10k. Production set up typically does not have slow follower availability concerns and functional testing is keep using 2000 for quite long time.

etcd/tests/functional/functional.yaml

Line 32 in ad03f20

snapshot-count: 2000

The benefits of making snapshot-count to 10k:

• etcd will consume less memory and less likely to out of memory killed
• etcd will comsume less disk space because the snapshot will release the WAL file locks and purgeFile can delete those old files before throwing error "wal: no space is left"
• avoid confusion like Kubeadm uses deprecated etcd default snapshot count kubernetes/kubernetes#100471

Any concerns about this ^ @serathius @ahrtr @ptabor @spzala

Reference

@gyuho
TLDR; I recommend lowering the default value in etcd instead of growing the value in k8s.

I think etcd snapshots are relatively 'lightweight'. The huge (100k+) value was justified when storev2 has been dumped completely with every snapshot. Currently the snapshots are ~10KB of data, mostly membership information, so 10k is a reasonable number IMHO. When storev2 is decomissioned (so etcd-3.6), I would consider checkpointing even more frequently (for faster recovery).

• keep etcd default value for snapshot count in kubernetes the same with etcd community kubernetes/kubernetes#100475 (comment)

• etcdserver: increase snapshot to 100,000 introduced in v3.2.x

• RCA of memory jump from etcd v3.1.x to etcd v3.2.x

• Rethink of raft log compaction (a more dynamic and graceful way to compact raft log)

[chaochn47]

Possibly as part of minor release

https://docs.google.com/document/d/16XEGyPBisZvmmoIHSZzv__LoyOeluC5a4x353CX0SIM/edit#

[ptabor]

I agree that good reasons why it used to be 100'000 ended when storeV2 stopped containing key-value payload so cost of writing such snapshot is neglictable.

• I don't think we can make such change as part of 'patch' release.
• I think it's OK as part of minor release (v3.6)
  • We might put a warning it will be subject to change in 3.6 for those that does not set it explicitly?

[chaochn47]

Thanks @ptabor for the comments. This issue can be closed. I will follow up with @mitake about this #7162