optim.LoadStateDict from existing StateDict doesn't clone tensors

[shaltielshmid]

When I call optim.LoadStateDict from a state dictionary of an exiting optimizer, the tensors are copied by reference, so if the old optimizer gets disposed the tensors are invalid.

Sample code:

var lin1 = torch.nn.Linear(10, 10);

var optim1 = torch.optim.Adam(lin1.parameters());
var optim2 = torch.optim.Adam(lin1.parameters());
optim2.load_state_dict(optim1.state_dict());
optim1.Dispose();

torch.nn.functional.mse_loss(lin1.call(torch.rand(10)), torch.rand(10)).backward();
optim2.step();

Throws:

System.InvalidOperationException: 'Tensor invalid -- empty handle.'

[shaltielshmid]

This also causes issues with devices, if you copy a state dict from an optimizer that was on a different device.

For example:

var lin1 = torch.nn.Linear(10, 10);
var optim1 = torch.optim.Adam(lin1.parameters());
var sd = optim1.state_dict();

lin1.cuda();
var optim2 = torch.optim.Adam(lin1.parameters());
Console.WriteLine((optim2.state_dict().State[0] as Adam.State).exp_avg.device.type); // CUDA
optim2.load_state_dict(sd);
Console.WriteLine((optim2.state_dict().State[0] as Adam.State).exp_avg.device.type); // CPU