[SCRIPT] XLNet squad finetuning scripts

[zburning]

Description

add XLNet squad finetuning scripts.
add corresponding results.

Checklist

Essentials

• PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage
• Code is well-documented

Changes

• Feature1, tests, (and when applicable, API doc)
• Feature2, tests, (and when applicable, API doc)

Comments

• If this change is a backward incompatible change, why must this change be made.
• Interesting edge cases to note here

[codecov]

Codecov Report

Merging #1130 into master will increase coverage by 0.09%.
The diff coverage is 92.3%.

@@            Coverage Diff             @@
##           master    #1130      +/-   ##
==========================================
+ Coverage   88.29%   88.39%   +0.09%     
==========================================
  Files          67       71       +4     
  Lines        6316     6703     +387     
==========================================
+ Hits         5577     5925     +348     
- Misses        739      778      +39

Impacted Files	Coverage Δ
src/gluonnlp/data/transforms.py	83.05% <100%> (ø)	⬆️
src/gluonnlp/data/question_answering.py	100% <100%> (ø)	⬆️
src/gluonnlp/model/bert.py	92.65% <100%> (ø)	⬆️
src/gluonnlp/data/utils.py	85.79% <89.58%> (ø)	⬆️
src/gluonnlp/model/train/cache.py	97.67% <0%> (ø)	⬆️
src/gluonnlp/embedding/evaluation.py	95.79% <0%> (ø)	⬆️
src/gluonnlp/data/batchify/language_model.py	96.26% <0%> (ø)	⬆️
src/gluonnlp/model/translation.py	71.87% <0%> (ø)	⬆️
src/gluonnlp/model/train/language_model.py	88.51% <0%> (ø)	⬆️
src/gluonnlp/model/language_model.py	98.49% <0%> (ø)	⬆️
... and 21 more

[mli]

Job PR-1130/3 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1130/3/index.html

[eric-haibin-lin]

@leezu what do you think of adding 'XLNet' to gluonnlp.model API, and have a folder for squad finetuning with BERT and XLNet? We don't necessarily need to expose all intermediate block APIs (e.g. attention cell)

[mli]

Job PR-1130/4 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1130/4/index.html

[mli]

Job PR-1130/5 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1130/5/index.html

[mli]

Job PR-1130/6 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1130/6/index.html

[mli]

Job PR-1130/7 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1130/7/index.html

[mli]

Job PR-1130/8 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1130/8/index.html

[mli]

Job PR-1130/9 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1130/9/index.html

[mli]

Job PR-1130/10 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1130/10/index.html

[leezu]

There seems to be a bug

[2020-02-02T20:03:05.993Z] Traceback (most recent call last):

[2020-02-02T20:03:05.993Z]   File "./scripts/language_model/run_squad.py", line 200, in <module>

[2020-02-02T20:03:05.993Z]     xlnet_base, vocab, tokenizer = model.get_model(**get_model_params)

[2020-02-02T20:03:05.993Z]   File "/var/lib/jenkins/workspace/gluon-nlp-gpu-py3/scripts/language_model/transformer/model.py", line 62, in get_model

[2020-02-02T20:03:05.993Z]     return models[name](**kwargs)

[2020-02-02T20:03:05.993Z]   File "/var/lib/jenkins/workspace/gluon-nlp-gpu-py3/scripts/language_model/transformer/model.py", line 193, in xlnet_cased_l12_h768_a12

[2020-02-02T20:03:05.993Z]     ignore_extra=not kwargs.get('use_decoder', True))

[2020-02-02T20:03:05.993Z]   File "/var/lib/jenkins/workspace/gluon-nlp-gpu-py3/src/gluonnlp/model/utils.py", line 281, in _load_pretrained_params

[2020-02-02T20:03:05.993Z]     net.load_parameters(model_file, ctx=ctx, ignore_extra=ignore_extra, allow_missing=allow_missing)

[2020-02-02T20:03:05.993Z]   File "/var/lib/jenkins/workspace/gluon-nlp-gpu-py3/conda/gpu/py3/lib/python3.5/site-packages/mxnet/gluon/block.py", line 555, in load_parameters

[2020-02-02T20:03:05.993Z]     params[name]._load_init(loaded[name], ctx, cast_dtype=cast_dtype, dtype_source=dtype_source)

[2020-02-02T20:03:05.993Z]   File "/var/lib/jenkins/workspace/gluon-nlp-gpu-py3/conda/gpu/py3/lib/python3.5/site-packages/mxnet/gluon/parameter.py", line 310, in _load_init

[2020-02-02T20:03:05.993Z]     self._init_impl(data, ctx)

[2020-02-02T20:03:05.993Z]   File "/var/lib/jenkins/workspace/gluon-nlp-gpu-py3/conda/gpu/py3/lib/python3.5/site-packages/mxnet/gluon/parameter.py", line 359, in _init_impl

[2020-02-02T20:03:05.993Z]     self._data = [data.copyto(ctx) for ctx in self._ctx_list]

[2020-02-02T20:03:05.993Z]   File "/var/lib/jenkins/workspace/gluon-nlp-gpu-py3/conda/gpu/py3/lib/python3.5/site-packages/mxnet/gluon/parameter.py", line 359, in <listcomp>

[2020-02-02T20:03:05.993Z]     self._data = [data.copyto(ctx) for ctx in self._ctx_list]

[2020-02-02T20:03:05.993Z]   File "/var/lib/jenkins/workspace/gluon-nlp-gpu-py3/conda/gpu/py3/lib/python3.5/site-packages/mxnet/ndarray/ndarray.py", line 2632, in copyto

[2020-02-02T20:03:05.993Z]     return _internal._copyto(self, out=hret)

[2020-02-02T20:03:05.993Z]   File "<string>", line 27, in _copyto

[2020-02-02T20:03:05.993Z]   File "/var/lib/jenkins/workspace/gluon-nlp-gpu-py3/conda/gpu/py3/lib/python3.5/site-packages/mxnet/_ctypes/ndarray.py", line 107, in _imperative_invoke

[2020-02-02T20:03:05.993Z]     ctypes.byref(out_stypes)))

[2020-02-02T20:03:05.993Z]   File "/var/lib/jenkins/workspace/gluon-nlp-gpu-py3/conda/gpu/py3/lib/python3.5/site-packages/mxnet/base.py", line 255, in check_call

[2020-02-02T20:03:05.993Z]     raise MXNetError(py_str(_LIB.MXGetLastError()))

[2020-02-02T20:03:05.993Z] mxnet.base.MXNetError: [20:03:01] src/engine/threaded_engine.cc:333: Check failed: exec_ctx.dev_id < device_count_ (1 vs. 1) : Invalid GPU Id: 1, Valid device id should be less than device_count: 1

[mli]

Job PR-1130/11 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1130/11/index.html