[WIP][BUGFIX] Fix a data sampling bug in BERT pre-training

[eric-haibin-lin]

Description

@rich-junwang

Checklist

Essentials

• PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage
• Code is well-documented

Changes

• Feature1, tests, (and when applicable, API doc)
• Feature2, tests, (and when applicable, API doc)

Comments

• If this change is a backward incompatible change, why must this change be made.
• Interesting edge cases to note here

cc @dmlc/gluon-nlp-team

[codecov]

Codecov Report

Merging #977 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #977   +/-   ##
=======================================
  Coverage   89.83%   89.83%           
=======================================
  Files          67       67           
  Lines        6364     6364           
=======================================
  Hits         5717     5717           
  Misses        647      647

[mli]

Job PR-977/1 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-977/1/index.html