[Backport 2.x] Add a few dev guide based on observations

[Backport 2.x] Add a few dev guide based on observations #11119