Release Schedule

[NiklasGustafsson]

Adding a thread to discuss upcoming releases of TorchSharp.

[NiklasGustafsson]

0.95.4

I intend to make a release 0.95.4 to NuGet on December 3.

This will contain the DisposeScope addition, as well as some bug fixes and a couple of additional LR schedulers. If we can get the DataLoader PR completed, it may go in, too.

It will be the last NuGet release or TorchSharp this year, as I am taking vacation from December 6 onward. I will attend to critical bugs only during that time.

[NiklasGustafsson]

0.96.0

I intend to make another release to NuGet today, for version 0.96.0.

It contains fixes to a few bugs, and some breaking changes:

1. 'Module.Eval' is now 'Module.eval' and 'Module.Train' is 'Module.train'

2. 'Module.parameters' and 'Module.named_parameters()' now return IEnumerable instead of T[]. This is closer to the PyTorch API and potentially slightly more efficient, but it might break any API that assumes it's getting an array.

[NiklasGustafsson]

0.96.1

Another release will be coming up, after issues #510 and #495 are resolved. No date yet, but likely the week of February 14.

[NiklasGustafsson]

It's not likely to be happening this week. There's a huge PR coming up that needs to be reviewed, and that will take time.

[GeorgeS2019]

This is the PR

[NiklasGustafsson]

Don Syme created a release with some critical F#-related fixes. The big bug release will be 0.96.3 (I messed up the 0.96.2 release).

[NiklasGustafsson]

0.96.4

The big feature in this release will be support for .NET FX 4.7.2 and later.

That said, this release will come once open issues 551-559 are addressed. Volunteers to help with some of these issues would be appreciated, but it will get done no matter what.

Pytorch 1.11 has been released, but we will deal with that in a later release of TorchSharp.

[NiklasGustafsson]

I just sent 0.96.4 to NuGet. Please see the Release Notes for details.

[NiklasGustafsson]

I just release 0.96.5 to NuGet. The previous release had some issues when used in VS Code that were not caught before 0.96.4 was released. This release fixes those.

[NiklasGustafsson]

0.96.6 was just released to NuGet. There are a small number of bug fixes, and the addition of a few "standard" models under TorchVision.

[NiklasGustafsson]

NuGet Version 0.96.7

This release is now available on NuGet. Here's the summary of changes:

Dependency Changes:

This version integrates with the libtorch 1.10.0 backend. API updates to follow.

API Changes:

Strong name signing of the TorchSharp library to allow loading it in .NET Framework strongly name signed apps.

Added the 'META' device type, which can be used to examine the affect of shape from tensor operations without actually doing any computations.

Added a few methods from the torch.nn.utils namespace.

Add torch.stft() and torch.istft()

Fixed Bugs:

#567 pad missing the choice to fill at start or end

[NiklasGustafsson]

Just released to NuGet:

NuGet Version 0.96.8

Breaking Changes:

This release contains a fix to inadvertent breaking changes in 0.96.7, related to Tensor.str(). This fix is itself breaking, in that it breaks any code that relies on the order of
arguments to str() introduced in 0.96.7. However, since the pre-0.96.7 argument order makes more sense, we're taking this hit now rather than keeping the inconvenient order in 0.96.7.

Fixed Bugs:

#618 TorchSharp.Modules.Normal.sample() Expected all tensors [...]

#621 torch.roll missing

#629 Missing dependency in 0.96.7 calling TorchSharp.torchvision.datasets.MNIST

#632 gaussian_nll_loss doesn't work on GPU

API Changes:

Add torchaudio.datasets.YESNO().
Added torch.from_array() API to create a tensor from an arbitry-dimension .NET array.

Added torch.tensor() overloads for most common dimensions of .NET arrays: ndim = [1,2,3,4]

Added the most significant API additions from Pytorch 1.11.

Added two torchaudio APIs.

Added 'decimals' argument to Tensor.round()

Changed tensor.str() to undo the breaking change in 0.96.7

Added torch.std_mean()

[NiklasGustafsson]

Just released on NuGet:

NuGet Version 0.97.0

Fixed Bugs:

#653:Tensor.to(Tensor) doesn't change dtype of Tensor.

API Changes:

Add ability to load and save TorchScript modules created using Pytorch

Add torch.utils.rnn

Add torchvision.io

Add Tensor.trace() and torch.trace() (unrelated to torch.jit.trace)

Add Tensor.var and Tensor.var_mean

Add torchaudio.datasets.SPEECHCOMMANDS

Add torchaudio.Resample()

[NiklasGustafsson]

Just released on NuGet:

NuGet Version 0.97.1

This release is made shortly after 0.97.0, since it adresses a serious performance issue when creating large tensors from .NET arrays.

Fixed Bugs:

#670 Tensor allocation insanely slow for from_array()

API Changes:

RNN, LSTM, GRU support PackedSequence

Add element-wise comparison methods of torch class.

Fix clamp and (non)quantile method declarations

Implementing isnan()

Added torchaudio.models.Tacotron2()

[NiklasGustafsson]

Soon on NuGet:

NuGet Version 0.97.2

Breaking Changes:

This release contains a breaking change related to torch.tensor() and torch.from_array() which were not adhering to the semantics of the Pytorch equivalents (torch.from_numpy() in the case of torch.from_array()).

With this change, there will be a number of different APIs to create a tensor form a .NET array. The most significant difference between them is whether the underlying storage is shared, or whether a copy is made. Depending on the size of the input array, copying can take orders of magnitude more time in creation than sharing storage, which is done in constant time (a few μs).

The resulting tensors may be reshaped, but not resized.

// Never copy:
public static Tensor from_array(Array input)

// Copy only if dtype or device arguments require it:
public static Tensor frombuffer(Array input, ScalarType dtype, long count = -1, long offset = 0, bool requiresGrad = false, Device? device = null)
public static Tensor as_tensor(Array input,  ScalarType? dtype = null, Device? device = null)
public static Tensor as_tensor(Tensor input, ScalarType? dtype = null, Device? device = null)

// Always copy:
public static Tensor as_tensor(IList<<VARIOUS TYPES>> input,  ScalarType? dtype = null, Device? device = null)
public static Tensor tensor(<<VARIOUS TYPES>> input, torch.Device? device = null, bool requiresGrad = false)

Fixed Bugs:

#670 Better align methods for creating tensors from .NET arrays with Pytorch APIs. _This is the breaking change mentioned earlier.
#679 The default value of onesided or torch.istft() is not aligned with PyTorch

API Changes:

Added torch.nn.init.trunc_normal_
Added index_add, index_copy, index_fill
Added torch.frombuffer()
Added torch.fft.hfft2, hfftn, ihfft2, ihfftn
Adding SequentialLR to the collection of LR schedulers.
Add 'training' flag to functional dropout methods.
Add missing functions to torchaudio.functional
Adding TestOfAttribute to unit tests

[NiklasGustafsson]

Just released to NuGet:

NuGet Version 0.97.3

Fixed Bugs:

#694 torch.log10() computes torch.log()
#691 torch.autograd.backward()
#686 torch.nn.functional.Dropout() doesn't have the training argument.

API Changes:

Add repeat_interleave()
Add torch.broadcast_shapes()
Added meshgrid, mT, mH, and H
Added additional distributions.
Add dct and mu-law to torchaudio
Added torchvision -- sigmoid_focal_loss()
Update the arguments of dropout() in Tacotron2
Add static function for all(), any(), tile(), repeat_interleave().
Add an implementation of the ReduceLROnPlateau learning rate scheduler.

[NiklasGustafsson]

I release 0.97.4 to NuGet, but it's a bad release -- the Google.Protobuf dependency is not captured, so you have to install it manually in every project that uses Tensorboard. I will release 0.97.5 to fix it.

[NiklasGustafsson]

NuGet Version 0.97.5

Fixed Bugs:

#715 How to implement the following code

API Changes:

Add functional normalizations.
Added torch.utils.tensorboard.SummaryWriter. Support for scalars only.

[NiklasGustafsson]

NuGet Version 0.97.6

Breaking Changes:

This release changes TorchSharp.torchvision from a namespace to a static class. This will break any using directives that assumes that it is a namespace.

Fixed Bugs:

#719 ResNet maxpool
#730 Sequential.Add
#729 Changing torchvision namespace into a static class?

API Changes:

Adding 'append()' to torch.nn.Sequential
Adding torch.numel() and torch.version
Adding modifiable global default for tensor string formatting

[NiklasGustafsson]

NuGet Version 0.100.7

Even though the last release was only a week ago, I concluded that there was a critical mass of important bug fixes, especially making it work better with F# and fixing the iterable-style DataLoader issue.

Breaking Changes:

DataLoader should no longer be created using new -- instead, the overall pattern is followed, placing the classes in TorchSharp.Modules and the factories in the static class. This will break any code that creates a DataLoader, but can be fixed by:

1. Removing the new in new torch.utils.data.DataLoader(...)
2. Adding a using TorchSharp.Modules (C#) or open TorchSharp.Modules (F#) to files where DataLoader is used as a type name.

API Changes:

Adding an IterableDataset abstract class, and making TensorDataset derive from it.
Moving the DataLoader class to TorchSharp.Modules and adding DataLoader factories.
#1092: got error when using DataLoader
#1069: Implementation of torch.sparse_coo_tensor for sparse tensor creation
Renamed torch.nn.functional.SiLU -> torch.nn.functional.silu
Added a set of generic Sequential classes.

Bug Fixes:

#1083: Compiler rejects scalar operand due to ambiguous implicit conversion

[NiklasGustafsson]

NuGet Version 0.101.0

This is an upgrade to libtorch 2.1.0. It also moves the underlying CUDA support to 12.1 from 11.7, which means that all the libtorch-cuda-* packages have been renamed. Please update your CUDA driver to one that support CUDA 12.1 before moving to this version of TorchSharp.

API Changes:

Enhanced Module.load function to return matching status of parameters in non-strict mode via an output dictionary.
Introduced attribute-based parameter naming for module state dictionaries, allowing custom names to override default field names.

[NiklasGustafsson]

NuGet Version 0.101.1

This is a fast-follower bug fix release, addressing persistent issues with stability of using TorchScript from TorchSharp. Given the significance of this fix, a same-day release is made.

Bug Fixes:

#1047 Torchscript execution failures (hangs, access violation, Fatal error. Internal CLR fatal error. (0x80131506) )

[NiklasGustafsson]

NuGet Version 0.101.2

API Changes:

Added extension method ScalarType.ElementSize() to get the size of each element of a given ScalarType.
Added methods for loading and saving individual tensors with more overloads.
Added 'persistent' flag to register_buffer()

[NiklasGustafsson]

NuGet Version 0.101.3

Breaking Changes:

The base OptimizerState class was modified and includes two changes:

1. Custom optimizer state objects derived from OptimizerState must now explicitly pass the related torch.nn.Parameter object to the OptimizerState base constructor to maintain correct linkage.
2. Custom state objects must implement an Initialize function. This function is responsible for initializing the properties of the state. Note that this function can be called as a re-intialization, so proper disposal of the previous tensor objects should be handled.

API Changes:

Introduced InferenceMode, a block-based scoping class for optimizing TorchSharp model inference by disabling gradient computation and enhancing performance.
Added Tensor.to_type() conversion aliases for short, half, bfloat16, cfloat, and cdouble.
Added Module.to() conversion aliases for all the scalar types.
All distribution classes now implement IDisposable.

Bug Fixes:

#1154 : mu_product was not initialized in NAdam optimizer
#1170 : Calling torch.nn.rnn.utils.pad_packed_sequence with a CUDA tensor and unsorted_indices threw an error
#1172 : optim.LoadStateDict from an existing StateDictionary updated to make sure to copy value and to the right device.
#1176 : When specific Optimizers load in a conditional tensor, made sure to copy to the right device.
#1174 : Loading CUDA tensor from stream threw an error
#1179 : Calling Module.to() with the ParameterList and ParameterDict module didn't move the parameters stored in the field.
#1148 : Calling Module.to() shouldn't be differentiable
#1126 : Calling ScriptModule.to() doesn't move attributes
#1180 : Module.to(ScalarType) has restrictions in PyTorch which aren't restricted in TorchSharp.

[NiklasGustafsson]

NuGet Version 0.101.4

A fast-follow release addressing a regression in v0.101.3

Bug Fixes:

#1185 : Incomplete transfer of module to device (only with 0.101.3)

[NiklasGustafsson]

NuGet Version 0.101.5

Another release just before 2024. There will be no more releases in 2023. It addresses and important bug in Module.to()

Bug Fixes:

#1191 : Having trouble moving a module from one GPU to another with gradients.

[NiklasGustafsson]

NuGet Version 0.101.6

API Changes:

#1223: Missing prod function torch.prod or a.prod() where a is Tensor
#1201: How to access the attributes of a model?
#1094: ScriptModule from Stream / ByteArray
#1149: Implementation for torch.autograd.functional.jacobian to compute Jacobian of a function
Implemenation for a custom torch.autograd.Function class

Bug Fixes:

#1198: CUDA not available when calling backwards before using CUDA
#1200: Bugs in torch.nn.AvgPool2d and torch.nn.AvgPool3d methods.

[NiklasGustafsson]

NuGet Version 0.102.0

This release upgrades the libtorch backend to v2.2.0.

Breaking Changes:

The Ubuntu builds are now done on a 22.04 version of the OS. This may (or may not) affect TorchSharp use on earlier versions.
The default value for the end_factor argument in the constructor for LinearLR was changed to 1.0 to match PyTorch.
Any code that checks whether a device is 'CUDA' and does something rather than checking that it isn't 'CPU' will now fail to work, since there is now support for the 'MPS' device on MacOS.

API Changes:

#652: Apple Silicon support .
#1219: Added support for loading and saving tensors that are >2GB.

Bug Fixes:

Fixed LinearLR scheduler calculation with misplaced parentheses.
Added get_closed_form_lr to scheduler to match PyTorch behavior when specifying epoch in .step()

[NiklasGustafsson]

NuGet Version 0.102.1

Breaking Changes:

The kernelSize parameter in the function and class of AvgPool1D was renamed to kernel_size to match PyTorch naming.
The stride parameter in the torch.nn.functional.avg_pool1d call now defaults to kernelSize instead of 1, to match the PyTorch behavior.

Bug Fixes:

module.load_state_dict() throws error for in-place operation on a leaf variable that requires grad.
#1250 cstr and npstr for 0d tensors
#1249 torch.nn.functional.avg_pool1d is not working correctly
module.load() with streams which don't read the requested # of bytes throws error.
#1246 Issue running in notebook on Apple Silicon

[NiklasGustafsson]

NuGet Version 0.102.2

This was a fast-follower release based on a pretty important bug fix that caused memory leaks.

Bug Fixes:

#1257 InverseMelScale in NewDisposeScope doesn't dispose tensors

[NiklasGustafsson]

NuGet Version 0.102.3

Breaking Changes:

The 'paddingMode' parameter of convolution has been changed to 'padding_mode', and the 'outputPadding' is now 'output_padding'.

API Changes:

#1243 fuse_conv_bn_weights and fuse_linear_bn_weights are added.
#1274 ConvTranspose3d does not accept non-uniform kernelSize/stride values.

[NiklasGustafsson]

NuGet Version 0.102.4

Breaking Changes:

Correct torch.finfo. (torch.set_default_dtype, Categorical.entropy, _CorrCholesky.check, Distribution.ClampProbs, FisherSnedecor.rsample, Gamma.rsample, Geometric.rsample, distributions.Gumbel, Laplace.rsample, SigmoidTransform._call and SigmoidTransform._inverse are influenced.)

API Changes:

#1284 make torch.unique and torch.unique_consecutive public.

[NiklasGustafsson]

NuGet Version 0.102.5

Breaking Changes:

torchvision.dataset.MNIST will try more mirrors. The thrown exception might be changed when it fails to download MNIST, FashionMNIST or KMNIST.
ObjectDisposedException will now be thrown when trying to use a disposed dispose scopes.
The constructor of dispose scopes is no longer public. Use torch.NewDisposeScope instead.

API Changes:

#1317 How to set default device type in torchsharp.
#1314 Grant read-only access to DataLoader attributes
#1313 Add 'non_blocking' argument to tensor and module 'to()' signatures.
#1291 Tensor.grad() and Tensor.set_grad() have been replaced by a new property Tensor.grad.
A potential memory leak caused by set_grad has been resolved.
Include method of dispose scopes has been removed. Use Attach instead.
Two more Attach methods that accepts IEnumerable<IDisposable>s and arrays as the parameter have been added into dispose scopes.
A new property torch.CurrentDisposeScope has been added to provide the ability to get the current dispose scope.
Add module hooks that take no input/output arguments, just the module itself.

Bug Fixes:

#1300 Adadelta, Adam and AdamW will no longer throw NullReferenceException when maximize is true and grad is null.
torch.normalwill now correctly return a leaf tensor. New optionsdisposeBatchanddisposeDatasethave been added intoDataLoader`.
The default collate functions will now always dispose the intermediate tensors, rather than wait for the next iteration.

Bug Fixes:

TensorDataset will now keep the aliases detached from dispose scopes, to avoid the unexpected disposal.
DataLoaderEnumerator has been completely rewritten to resolve the unexpected shuffler disposal, the ignorance of drop_last, the incorrect count of worker, and the potential leak cause by multithreading.
#1303 Allow dispose scopes to be disposed out of LIFO order.

[NiklasGustafsson]

NuGet Version 0.103.0

Move to libtorch 2.4.0.
Jettison MacOS / Intel support (libtorch no longer supports it).
Accumulated bug fixes.

Breaking Changes:

When creating a tensor from a 1-D array, and passing in a shape, there is now an ambiguity between the IList and Memory overloads of torch.tensor(). The ambiguity is resolved by removing the dimensions argument if it is redundant, or by an explicit cast to IList if it is not.

Bug Fixes:

#1359 torch.nn.functional.l1_loss computes a criterion with the MSE, not the MAE.
#1334 MultivariateNormal.log_prob() exception in TorchSharp but works in pytorch.

API Changes:

#1326 Allow arrays used to create tensors to be larger than the tensor. Create tensors from a Memory instance.