[Parallel] SQuAD multi-GPU script

[eric-haibin-lin]

Description

@szhengac

Checklist

Essentials

• PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage
• Code is well-documented

Changes

• Feature1, tests, (and when applicable, API doc)
• Feature2, tests, (and when applicable, API doc)

Comments

• If this change is a backward incompatible change, why must this change be made.
• Interesting edge cases to note here

[codecov]

Codecov Report

Merging #1079 into master will increase coverage by 0.04%.
The diff coverage is 95.23%.

@@            Coverage Diff             @@
##           master    #1079      +/-   ##
==========================================
+ Coverage   88.25%   88.29%   +0.04%     
==========================================
  Files          67       67              
  Lines        6291     6316      +25     
==========================================
+ Hits         5552     5577      +25     
  Misses        739      739

Impacted Files	Coverage Δ
src/gluonnlp/model/bert.py	92.65% <95.23%> (+0.47%)	⬆️
src/gluonnlp/data/utils.py	85.79% <0%> (-0.6%)	⬇️
src/gluonnlp/model/transformer.py	91.66% <0%> (+0.32%)	⬆️
src/gluonnlp/data/transforms.py	83.05% <0%> (+0.32%)	⬆️
src/gluonnlp/model/attention_cell.py	92.17% <0%> (+0.55%)	⬆️

[szhengac]

I would not recommend using horovod for finetuning task at this time, as the pre-built newest mxnet does not work with horovod at the moment. The user has to use older version of mxnet or compile it from source.

…

On Sun, 22 Dec 2019 at 21:36, codecov[bot] ***@***.***> wrote: Codecov <https://codecov.io/gh/dmlc/gluon-nlp/pull/1079?src=pr&el=h1> Report ❗️ No coverage uploaded for pull request head ***@***.***). Click here to learn what that means <https://docs.codecov.io/docs/error-reference#section-missing-head-commit> . The diff coverage is n/a. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1079?email_source=notifications&email_token=AA6GZVDJYNHNMNFYISHERKTQ2BE7HA5CNFSM4J6P5MWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHQIJIY#issuecomment-568362147>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA6GZVCU2GFEVSC2S2DKWJDQ2BE7HANCNFSM4J6P5MWA> .

[leezu]

Nightly builds are fixed and work with horovod since 2 weeks. See apache/mxnet#16193

But note that nightly builds are not published on pypi anymore.

[eric-haibin-lin]

@szha @ptrendx mxnet pypi limit was increased. Will mxnet 1.6 stable release include the symbols needed by horovod?

[szha]

@eric-haibin-lin yes

[mli]

Job PR-1079/16 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1079/16/index.html

[mli]

Job PR-1079/17 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1079/17/index.html

[mli]

Job PR-1079/18 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1079/18/index.html

[mli]

Job PR-1079/19 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1079/19/index.html

[mli]

Job PR-1079/20 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1079/20/index.html

[mli]

Job PR-1079/21 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1079/21/index.html