Automatically saving run files in pt.Experiment()

[cmacdonald]

Is your feature request related to a problem? Please describe.

I want to save results as I evaluate them

Describe the solution you'd like

Add a files= kwarg to pt.Experiment() (or filenames=)

pt.Experiment(
  systems=[bm25, dph],
  ...
  files=['bm25.res.gz', 'dph.res.gz']
)

Describe alternatives you've considered

1. Current neatest solution - appending a transformer to save results:

def save(filename) -> TransformerBase:
  def _save(df):
    pt.io.write_results(df, filename)
    return df
  return pt.apply.generic(_save)

pt.Experiment(
  systems=[bm25 >> save('bm25.res.gz') , dph  >> save('dph.res.gz')],
  ...
)

Pros:

• Can be done now without needing any change

Cons:

• Klunky to do for each

2. Save to directory

pt.Experiment(
  systems=[bm25, dph],
  ...
  save_dir='/dir/to/save/',
  names=['bm25', 'dph']
)

would create files named bm25.res.gz and dph.res.gz in /dir/to/save/

Pros:

• dont have to construct filenames manually
  Cons:
• seems inelegant

Suggested by @albertoueda

[cmacdonald]

I think we should revisit this. For reproducibility purposes, e.g. in papers, its important to save runs as they are created.
My current favourite is:

2. Save to directory

pt.Experiment(
  systems=[bm25, dph],
  ...
  save_dir='/dir/to/save/',
  names=['bm25', 'dph']
)

A further kwarg could define the saving/caching behaviour, e.g.

• save='overwrite' - for always replace the files
• save='reuse' - for always use files that already exist

(what would be the default)?

[cmacdonald]

Now addressed in #247