Computing train metrics at a given frequency

[Ghelfi]

🚀 Feature Request

Add the possibility to compute metrics on the train dataset with a given frequency and avoid computing these every steps

Motivation

Having the possibility to compute training metrics is great and allow a good control on what is happening at this stage. Some metrics can be quite compute intensive and we might want to compute these on a lower frequency to reduce to compute needed.
This should come with a default behaviour falling back to the current one: evaluating the training metrics for each batch.

Implementation

There is a eval_interval argument for the when to run evaluation and I propose to keep the same logic with a train_metric_interval to decide when to compute training metrics.

Happy to have a look and propose an implementation.

[mvpatel2000]

Hm... this is a bit tricky since this would affect all metrics...

Two proposed workarounds:

1. Store metric separately in callback, which easily gives control over frequency
2. Have code in metric to control frequency and ignore some calls (not ideal)

What do you think of alternatives? Not strongly opposed to proposal, but my regularizer is to avoid adding trainer args as much as possible