Adding Mixup and Cutmix #4379

datumbox · 2021-09-07T18:48:32Z

Partially resolves #3817, resolves #4281

I'm adding Mixup and Cutmix in References instead in the Transforms to allow the investigation of the new Transforms API to conclude. The transforms were tested using unit-tests which can be found in earlier commit. Prior to moving them in the Transforms area, we should consider introducing a base-class and eliminate duplicate code between the two implementations.

datumbox

Highlighting some "interesting" parts below to provide context. The PR is not yet ready.

torchvision/transforms/transforms.py

datumbox · 2021-09-08T11:19:44Z

torchvision/transforms/transforms.py

+    def __init__(self, num_classes: int,
+                 p: float = 1.0, mixup_alpha: float = 1.0,
+                 cutmix_p: float = 0.5, cutmix_alpha: float = 0.0,
+                 label_smoothing: float = 0.0, inplace: bool = False) -> None:


I understand that the choice of offering inplace support won't make everyone happy. The reason I decided to support it is:

It is more performant in terms of memory and speed.

Most the tensor-only transforms on torchvision typically support inplace operations (see Normalize and RandomErasing).

The ClassyVision version of this method supported inplace operations and thus to ensure we are equally performant we should support it as well.

I'm still a bit skeptical about adding this in-place operation, as most of the operators in here are actually not performing any in-place operation, so the gains of having an inplace flag might not be that large. But we might need to benchmark to be able to know for sure

I don't see why we can't support an in-place operation if one can be implemented correctly. For now I plan to move the entire implementation on the references script so that we have time to think this through before we commit to the API, so we don't have to solve this now. But overall I think inplace is a valid optimization and provided it can be implemented properly, there is no reason not do get the extra speed improvements.

As discussed offline with @fmassa we are going to review the use of inplace before moving this from references to transforms.

torchvision/transforms/transforms.py

test/test_transforms_tensor.py

fmassa

Thanks a ton for the PR!

Adding a few high-level comments, let me know what you think

torchvision/transforms/transforms.py

fmassa · 2021-09-09T12:31:48Z

torchvision/transforms/transforms.py

+    def __init__(self, num_classes: int,
+                 p: float = 1.0, mixup_alpha: float = 1.0,
+                 cutmix_p: float = 0.5, cutmix_alpha: float = 0.0,
+                 label_smoothing: float = 0.0, inplace: bool = False) -> None:


I'm still a bit skeptical about adding this in-place operation, as most of the operators in here are actually not performing any in-place operation, so the gains of having an inplace flag might not be that large. But we might need to benchmark to be able to know for sure

datumbox

I took care of splitting the methods. Below I highlight other interesting bits that were not there on the previous review.

torchvision/transforms/transforms.py

datumbox · 2021-09-13T17:40:04Z

torchvision/transforms/transforms.py

+    """
+
+    def __init__(self, num_classes: int,
+                 p: float = 0.5, alpha: float = 1.0,


I changed the default to p=0.5 to keep it consistent with other transforms.

datumbox · 2021-09-13T17:41:47Z

torchvision/transforms/transforms.py

+        return s.format(**self.__dict__)
+
+
+class RandomCutmix(torch.nn.Module):


There is some code duplication and thus bits that can be shared across the 2 classes, but this will be fixed on the new API with proper class inheritance.

datumbox · 2021-09-13T17:58:54Z

references/classification/train.py

@@ -273,6 +285,8 @@ def get_args_parser(add_help=True):
    parser.add_argument('--label-smoothing', default=0.0, type=float,
                        help='label smoothing (default: 0.0)',
                        dest='label_smoothing')
+    parser.add_argument('--mixup-alpha', default=0.0, type=float, help='mixup alpha (default: 0.0)')
+    parser.add_argument('--cutmix-alpha', default=0.0, type=float, help='cutmix alpha (default: 0.0)')


I'm exposing here very few options (I'm using hardcoded p params). Keeping it simple until we use it in models to see what we want to support.

@datumbox nn.CrossEntropyLoss dose not work when you use mixup or cutmix, because the traget shape is (N, K), rather (N,)

I think this was added at pytorch/pytorch#63122

This should be available on the latest stable version of pytorch. See doc.

@datumbox ok, thanks.

fmassa

LGTM, thanks!

I've made a few minor comments, none of which are merge-blocking.

torchvision/transforms/transforms.py

test/test_transforms.py

fmassa · 2021-09-15T11:09:37Z

references/classification/train.py

@@ -165,10 +166,21 @@ def main(args):
    train_dir = os.path.join(args.data_path, 'train')
    val_dir = os.path.join(args.data_path, 'val')
    dataset, dataset_test, train_sampler, test_sampler = load_data(train_dir, val_dir, args)
+
+    collate_fn = None
+    num_classes = len(dataset.classes)


not for now, but this exposes a limitation of our current datasets, which is that we don't consistently enforce a way of querying the number of classes in a dataset. The dataset refactoring work from @pmeier will address this

With #4432, you will be able to do

info = torchvision.datasets.info(name) info.categories

where categories is a list of strings in which the index corresponds to the label.

torchvision/transforms/transforms.py

test/test_transforms_tensor.py

Summary: * Add RandomMixupCutmix. * Add test with real data. * Use dataloader and collate in the test. * Making RandomMixupCutmix JIT scriptable. * Move out label_smoothing and try roll instead of flip * Adding mixup/cutmix in references script. * Handle one-hot encoded target in accuracy. * Add support of devices on tests. * Separate Mixup from Cutmix. * Add check for floats. * Adding device on expect value. * Remove hardcoded weights. * One-hot only when necessary. * Fix linter. * Moving mixup and cutmix to references. * Final code clean up. Reviewed By: datumbox Differential Revision: D31268036 fbshipit-source-id: 6a73c079d667443da898e3b175b88978b24d52ad

datumbox added 2 commits September 7, 2021 18:15

Add RandomMixupCutmix.

2de7cfc

Add test with real data.

55aedd3

datumbox added module: transforms new feature labels Sep 7, 2021

facebook-github-bot added the cla signed label Sep 7, 2021

datumbox marked this pull request as draft September 7, 2021 18:48

Use dataloader and collate in the test.

33d9575

oke-aditya mentioned this pull request Sep 8, 2021

[RFC] New Augmentation techniques in Torchvison #3817

Open

17 tasks

Making RandomMixupCutmix JIT scriptable.

c15dce0

datumbox commented Sep 8, 2021

View reviewed changes

datumbox mentioned this pull request Sep 8, 2021

torch.distributions.normal.Normal is not JIT supported pytorch/pytorch#29843

Open

fmassa reviewed Sep 9, 2021

View reviewed changes

datumbox mentioned this pull request Sep 9, 2021

[RFC] TorchVision with Batteries included - Phase 1 #3911

Closed

16 tasks

datumbox and others added 2 commits September 9, 2021 18:32

Move out label_smoothing and try roll instead of flip

6f2ebea

Merge branch 'main' into transforms/mixupcutmix

c1bc525

datumbox force-pushed the transforms/mixupcutmix branch 5 times, most recently from 0d115af to c4ca8c9 Compare September 9, 2021 19:22

Adding mixup/cutmix in references script.

67acd89

datumbox force-pushed the transforms/mixupcutmix branch from c4ca8c9 to 67acd89 Compare September 9, 2021 19:23

datumbox added 2 commits September 9, 2021 21:57

Handle one-hot encoded target in accuracy.

544967e

Add support of devices on tests.

0e128f3

datumbox requested a review from fmassa September 13, 2021 17:16

Separate Mixup from Cutmix.

3f19902

datumbox force-pushed the transforms/mixupcutmix branch from bd72c0a to 3f19902 Compare September 13, 2021 17:29

Add check for floats.

78e8605

datumbox commented Sep 13, 2021

View reviewed changes

datumbox added 2 commits September 13, 2021 19:21

Adding device on expect value.

de9fa07

Remove hardcoded weights.

3f212fe

fmassa approved these changes Sep 15, 2021

View reviewed changes

datumbox commented Sep 15, 2021

View reviewed changes

test/test_transforms_tensor.py Outdated Show resolved Hide resolved

datumbox marked this pull request as ready for review September 15, 2021 12:49

datumbox and others added 3 commits September 15, 2021 13:58

One-hot only when necessary.

33c2973

Fix linter.

9abb18b

Merge branch 'main' into transforms/mixupcutmix

b5bf8fc

datumbox added module: reference scripts and removed module: transforms labels Sep 15, 2021

datumbox changed the title ~~[WIP] Adding Mixup and Cutmix~~ Adding Mixup and Cutmix Sep 15, 2021

datumbox force-pushed the transforms/mixupcutmix branch from 4724bfe to 4a89568 Compare September 15, 2021 14:05

Moving mixup and cutmix to references.

eb932b9

datumbox force-pushed the transforms/mixupcutmix branch from 4a89568 to eb932b9 Compare September 15, 2021 14:17

Final code clean up.

b548b7a

datumbox force-pushed the transforms/mixupcutmix branch from 18a2969 to b548b7a Compare September 15, 2021 16:08

Merge branch 'main' into transforms/mixupcutmix

e3be92b

datumbox merged commit c8e3b2a into pytorch:main Sep 15, 2021

datumbox deleted the transforms/mixupcutmix branch September 15, 2021 17:32

datumbox mentioned this pull request Sep 20, 2021

Update reference scripts to use the "Batteries Included" utils #4281

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding Mixup and Cutmix #4379

Adding Mixup and Cutmix #4379

datumbox commented Sep 7, 2021 •

edited

Loading

datumbox left a comment

datumbox Sep 8, 2021

fmassa Sep 9, 2021

datumbox Sep 9, 2021

datumbox Sep 10, 2021

fmassa left a comment

fmassa Sep 9, 2021

datumbox left a comment

datumbox Sep 13, 2021

datumbox Sep 13, 2021

datumbox Sep 13, 2021

xiaohu2015 Nov 11, 2021

datumbox Nov 11, 2021

xiaohu2015 Nov 11, 2021

fmassa left a comment

fmassa Sep 15, 2021

pmeier Sep 16, 2021

		return s.format(**self.__dict__)


		class RandomCutmix(torch.nn.Module):

Adding Mixup and Cutmix #4379

Adding Mixup and Cutmix #4379

Conversation

datumbox commented Sep 7, 2021 • edited Loading

datumbox left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fmassa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

datumbox left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fmassa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

datumbox commented Sep 7, 2021 •

edited

Loading