Add more documentation

[blefaudeux]

🚀 Feature

• Surface all the features exposed by fairscale
• Add code snippets
• Add benchmark examples
• Add links towards other projects which are using fairscale already
• Add some thoughts on when and where use the different methods exposed by fairscale

Motivation

Fairscale is hard to grok from the outside, lower the barrier of entry

Pitch

Easier to onboard people, easier to increase traction, clean things up

Alternatives

live in a cave

Additional context

[blefaudeux]

cc @msbaines @froody @min-xu-ai

[froody]

We should also add guidance around picking optimal parameters. For example with gpipe, the right value for the 'chunks' parameter depends on the model size.

[VitaliyLi]

would be also nice to add a discussion on when to use specific features, ie when to use pipelining and when to use tensor parallelism

[blefaudeux]

We should also add guidance around picking optimal parameters. For example with gpipe, the right value for the 'chunks' parameter depends on the model size.

some frameworks expose the per param group flops, that's an approximation (ie not all flops take the same time to compute), but would that still be a useful heuristic, or should the size (in memory) be taken into account ? for my general understanding

[blefaudeux]

closed for now, to be revived if need be