Save/Load surpyval

[lisandrojim]

Hello,

I am trying to save locally a survival model. But I get an error. Below an example:

import surpyval as surv
import numpy as np
import pickle
from joblib import dump

np.random.seed(10)
x = surv.Weibull.random(50, 30., 9.)
model = surv.Weibull.fit(x)
results = {'model':model}

# Using pickle
pickle.dump({'model':model},open('surpyval_model',"wb"))
# Using dump
dump(model,'surpyval_model.joblib')

None of the methods work. These are the errors I get:

Using pickle
AttributeError: Can't pickle local object 'bounds_convert.<locals>.transform'
Using dump
PicklingError: Can't pickle <function bounds_convert.<locals>.transform at 0x7f98c4178a60>: it's not found as surpyval.parametric.fitters.bounds_convert.<locals>.transform

Any advice?

[derrynknife]

With a simple Weibull distribution it might be best to simply keep the parameters in a dict:

alpha, beta = model.params

model_dict = {
  'alpha': alpha,
  'beta': beta
}

This dictionary can then be pickled or json.dumped. To create the model again all you need do is:

model = surv.Weibull.from_params([model_dict['alpha'], model_dict['beta']])

[derrynknife]

I imagine you would prefer a more general solution?

[lisandrojim]

Yes, indeed I was looking for a more general solution. I tried:

• What you suggested. Saving the hyperparameters and re-build the model later with the function "from_params". The problem with this is that if you want, for example, to evaluate your model based on the aic criteria, it throws an error because it does not have the input data.
• The second I tried was to rebuild the model using the function "fit", giving input "x", "c", and "fixed". The problem is that this did not work for all the reliability models.

I hope there is a more generic solution where one can save/load any type of model in the future. I think this is useful.

[derrynknife]

Thanks @lisandrojim, I will add a general save and load feature for models.

Until then this will work (if using MLE):

np.random.seed(10)
x = surv.Weibull.random(50, 30., 9.)
model = surv.Weibull.fit(x)

alpha, beta = model.params

model_dict = {
  'alpha': alpha,
  'beta': beta,
  '_neg_ll': model._neg_ll,
  'hess_inv': model.hess_inv.tolist()
}

Then to recover the model:

model = surv.Weibull.from_params([model_dict['alpha'], model_dict['beta']])
model.hess_inv = np.array(model_dict['hess_inv'])
model._neg_ll = model_dict['_neg_ll']

If you do it this way the aic and bic methods will work

I'll close this issue when a general save/load feature is working.

[derrynknife]

@lisandrojim, I just pushed a version, 0.10.7, that includes to_dict() and from_dict() methods. E.g.:

x = surv.Weibull.random(10, 10, 2)
model = surv.Weibull.fit(x, how='MPP')
model_dict = model.to_dict()

recovered_model = surv.Parametric.from_dict(model_dict)

If you want to save it to a file you can pickle it or json dump it, up to you.

Let me know if it suits.

[lisandrojim]

@derrynknife thanks for implementing this preliminary solution. I tried it out and it works to rebuild the model from the dictionary. However, the recovered model lacks methods, such as .aic(), here is an example:

import surpyval as surv
x = surv.Weibull.random(10, 10, 2)
model = surv.Weibull.fit(x, how='MPP')
model_dict = model.to_dict()

recovered_model = surv.Parametric.from_dict(model_dict).aic()

The error:

AttributeError: 'Parametric' object has no attribute '_neg_ll'

[lisandrojim]

Hello @derrynknife, I found a solution to this issue. It is solved by using dill instead of pickle. For example:

import dill
import surpyval as surv

x = surv.Weibull.random(10, 10, 2)
model = surv.Weibull.fit(x, how='MPP')

your_file_location = '/Desktop/FileName'
dill.dump(model, file = open(your_file_location, "wb"))
recovered_model = dill.load(open(your_file_location, "rb"))