Model averaging of top trials

[cooperlab]

We could average top trials or compose an ensemble of sensitive and specific models for prediction. This should improve accuracy and also enables calculation of uncertainties. Perhaps this cannot be generalized for all applications and should be handled in the application libraries instead.

[cooperlab]

@lawrence-chillrud one idea is to accept a list of checkpoint directories as inputs and build and save a new model that returns a simple average. This could be similar to the top_k function.

If you are thinking more about a Bayesian or trainable approach to averaging that is conditional on inputs we would need to define what that is.

[cooperlab]

Given a validation set, we could identify models that have less correlated outputs for inclusion in the ensemble.