Patch 2478

[ahoblitz]

Fixes #2478

Description

Use nn.Transformer.generate_square_subsequent_mask when src_mask is None (Had to run to around ~30 epochs to get somewhat sensible output

Checklist

• The issue that is being fixed is referred in the description (see above "Fixes #ISSUE_NUMBER")
• Only one issue is addressed in this pull request
• Labels from the issue that this PR is fixing are added to this pull request
• No unnecessary issues are included into this pull request.

cc @pytorch/team-text-core @Nayef211 @sekyondaMeta @svekars @carljparker @NicolasHug @kit1980 @subramen

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/tutorials/2662

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 3d1cafc with merge base 16e4f2a ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[albanD]

@drisspg can you double check this please?

[drisspg]

I think I am okay with this but can you add some to the description saying that this is doing Causal Language Modeling and maybe a comment as to why we need this mask

[ahoblitz]

@albanD @drisspg I added some text description to explain the need for generating a square causal mask for the sequence