Integration of TrivialAugment with the current AutoAugment Code

[SamuelGabriel]

Dear Torchvision Maintainers,

Thank you for maintaining this very helpful library!
I am the author of the TrivialAugment paper (https://arxiv.org/abs/2103.10158) which I will present as an oral at the upcoming ICCV. The paper proposes a trivial baseline for automatic augmentation methods, which outperforms setups like AutoAugment or RandAugment on all evaluated image classification tasks. Since, it is a trivial baseline, I have the impression it is something solid to start from before thinking about augmentations. Therefore, I believe it makes sense to have it in the vision library. What do you think?

I implemented an initial proposal, including tests, that already uses the even stronger augmentation space setting, the results for which we publish in the update of the paper (ta_wide).

What I did in this PR:

• Abstract common functionality away from AutoAugment
• Implement TrivialAugment
• Make both TrivialAugment and AutoAugment pass the test/test_transforms_tensor.py

What I did not do so far, but offer to do:

• Write documentation
• Test this implementation in a real-world training

Best regards,

Sam

cc @vfdev-5 @datumbox

[facebook-github-bot]

Hi @SamuelGabriel!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@fb.com. Thanks!

[facebook-github-bot]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks!

[SamuelGabriel]

I'd love some comment on this / review. For example by you, @datumbox?

[datumbox]

@SamuelGabriel Apologies for the delayed response. I was OOO. Now I'm back and trying to catch up. I'll get back to you as soon as possible.

[SamuelGabriel]

Thank You! No worries, I guess new features are not very urgent compared to other things. :)

[datumbox]

@SamuelGabriel They are important! I was just away for sometime and now try to catch up. This is on the top of my list so I will try to get back to you within the next week.

BTW I see that there are a few conflicts with master. Do you prefer to resolve now or wait for me to review first? For full transparency I didn't have the chance to check out the PR yet.

[SamuelGabriel]

After fixing the diff and fixing the fixes, I have a mergeable version now. :) (Some wheels failed, but I believe this is due to something unrelated.)

[datumbox]

@SamuelGabriel Thanks a lot for the contribution.

Your PR is very well-written and I like the proposed refactoring of the original AutoAugment algorithm. The only comment I left below is related to a convention we have for Transforms that is not currently well documented but worth implementing if we decide to merge this.

As far as I can see, TrivialAugment is a fairly new method and your are one of the authors of the paper. I've not yet read the paper so I can't comment on the performance improvements it delivers, but typically in the TorchVision repo we avoid adding methods that are recently introduced. Typical criteria for assessing the inclusion of the technique are the number of references of the paper, the popularity of the technique in the community and whether it is considered mature enough so that we can provide a stable API.

@fmassa Could you share your thoughts on whether you think we should include the technique now or wait?

@SamuelGabriel it would be also very useful to hear your thoughts on how much mature you consider the algorithm and whether you believe you will iterate on it soon. Note that this PR does not have to be "all or nothing". I see that you already have a repo where you offer an official implementation of the algorithm, so perhaps we could modify this PR to include only the refactoring of AutoAugment that you deem necessary so that you can build upon it and extend it on your repo easier.

Looking forward to your input.

[SamuelGabriel]

@datumbox The TrivialAugment Algorithm is final. I don't foresee any changes to the algorithm in the future. It is the simplest working baseline I can think of, thus it is hard to iterate on.
I think it would be very valuable to offer to users in a standard library as it gives you an easy start on Augmentations, without all the complications that come with AutoAugment, but the same or better performance. You just use it, without specifying any hyper-parameters. I would not push for its inclusion if it wasn't such a trivial baseline.
Still, I totally get that you are not super excited to add an API for a newly published algorithm to torchvision. I am also happy to provide more experimental evidence for it working well in specific scenarios, but the paper already includes a lot of relevant scenarios for torch vision users: performance with standard models on CIFAR, SVHN and ImageNet, as well as some more exotic datasets in the updated (camera-ready) version from today.

I could also offer to add even more APIs: by implementing RandAugment alongside TrivialAugment, since the refactor allows to implement it with relative ease as well.

(But, of course, I am the author of TrivialAugment, and draw my motivation from people using my research findings: I am biased.)

[datumbox]

@SamuelGabriel Thanks for being transparent and for providing additional information. I agree with many of the points you raised.

I spoke also with Francisco and we would like a bit more time to investigate aka read the paper thoroughly, do some tests on our side etc. I currently got on the pipeline a few other urgent tasks, so I think I can get back to this in a couple of weeks. Does this work for you?

Concerning RandAugment, it's within my plans to add it (see #3817) but if you are interested in sending a separate PR that would be awesome. Just let me know if you start working on it to avoid duplication of effort.

Again thanks for sharing your contribution, let's keep the discussion open and decide what's best.

[SamuelGabriel]

@datumbox Great! Thanks, that works. I would need the AutoAugment refactorings of this pull-request, though, to make RandAugment simple, so I think it would be the easiest for me to simply add it to this PR when we know how we continue in a few weeks. Would that be good?

[datumbox]

@SamuelGabriel I think it would be best to separate the refactoring from the method so that we can merge them independently. Of course it also depends on your availability, but if you have bandwidth you are welcome to split this PR into two. This would allow us to merge the necessary refactoring ASAP while you implement RandAugment and we do the checks on TrivialAugment. Let me know. :)

[datumbox]

@SamuelGabriel I was hoping to get some clarifications concerning your ImageNet experiments at the paper:

• After preprocessing the image and applying AutoAugment, do you also apply CutOut?
• My understanding is that you don't recommend combining TA_wide with CutOut. How much Acc@1 do you lose on average by combining it?

Thanks!

[SamuelGabriel]

@datumbox Hi Vasilis,

Thank you for taking the time. Warning: I am actually on my honey moon this week so I might not reply as fast as I would like to until next week.

• After preprocessing the image and applying AutoAugment, do you also apply CutOut?

I apply CutOut only for non-imagenet experiments, just like the previous methods I compare to. As this is the standard practice.

• My understanding is that you don't recommend combining TA_wide with CutOut. How much Acc@1 do you lose on average by combining it?

No, that is not what I meant to say. I use the same pipeline as previous work. So TA_wide is used together with cutout for all CIFAR and SVHN experiments yielding SOTA scores. For ImageNet it is just not a common practice to use Cutout and was previously shown to not improve scores (I do not remember where tbh and saw I didn't put a citation there either). I believe experimenting on this with Imagenet is also hard as one would have to reimplement a baseline as well, since none of them use cutout. I could do it for comparing RA to TA anyways though, as I have an RA implementation ready.

Cheers,

Sam

[datumbox]

@SamuelGabriel Congratulations, best wishes to both of you! :)

No rush/worries, thanks for the information.

[datumbox]

@SamuelGabriel Thanks a lot for the contribution.

We did some checks on our end as well and we think that this is a good addition, especially for people who work on similar but different datasets than ImageNet. I should also note that your contribution and proposed approach helped us review the API of the augmentation code, so your contribution here is much bigger than just adding the TrivialAugment. Thank you!

As this PR contains also bug fixes for RandAugment, I'll merge this to get things going and we can follow up any further improvements on separate PRs. I just have one question for you below, please let me know.

[datumbox]

Should this also be 31 as per your comment at #4348 (comment)?

[SamuelGabriel]

Yes, it should be. I pushed a commit to change it to the underlying branch, but I am not sure what is the best way to get such a minor fix into main now, see SamuelGabriel@9df8f13

[datumbox]

No worries, I'll fix it on #4370.

[SamuelGabriel]

Great, Thanks!

[SamuelGabriel]

Oh, I am happy to see that TrivialAugment is now in main and to hear that I could help with the other automatic augmentation methods, too. :) Thank you for the community work that you do!

[datumbox]

@SamuelGabriel I forgot to share the findings of our runs with you.

The TrivialAugment was tested as a drop-in replacement of AutoAugment on the current training recipe of MobileNetV3 and it improved the accuracy by 0.1 acc points. Interestingly enough, using random-erasing (as with the baseline of AA) provides better results than without. We did 3 runs for each setup, below I report the results without weight averaging or EMA:

• AA: Acc@1 73.860 Acc@5 91.292
• TA: Acc@1 73.968 Acc@5 91.492

Note that the above experiment puts TA in a disadvantage comparing to AA because all other hyperparams in the recipe were tuned while using AA. The fact that TA beats it makes it quite impressive in my book.

[SamuelGabriel]

I wondered whether you didn't test TA before adding it to the library, I am happy you did and to see that you could use it so successfully out of the box, just like intended. :)