[MODEL] XLNet conversion scripts #866

leezu · 2019-08-07T17:34:44Z

Description

Add XLNet conversion scripts

Checklist

Essentials

PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage
Code is well-documented

Changes

Pretrained XLNet

Comments

This PR resolves #787

codecov · 2019-08-07T17:34:46Z

Codecov Report

Merging #866 into master will increase coverage by 15.86%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           master     #866       +/-   ##
===========================================
+ Coverage   73.87%   89.74%   +15.86%     
===========================================
  Files          67       67               
  Lines        6423     6423               
===========================================
+ Hits         4745     5764     +1019     
+ Misses       1678      659     -1019

Impacted Files	Coverage Δ
src/gluonnlp/data/utils.py	`74.04% <ø> (ø)`	⬆️
src/gluonnlp/data/batchify/embedding.py	`47.69% <0%> (-50.01%)`	⬇️
src/gluonnlp/vocab/subwords.py	`84.78% <0%> (-2.18%)`	⬇️
src/gluonnlp/data/batchify/batchify.py	`96.06% <0%> (+0.78%)`	⬆️
src/gluonnlp/data/transforms.py	`86% <0%> (+1.11%)`	⬆️
src/gluonnlp/data/question_answering.py	`100% <0%> (+1.66%)`	⬆️
src/gluonnlp/model/train/embedding.py	`87.17% <0%> (+2.56%)`	⬆️
src/gluonnlp/vocab/elmo.py	`96.66% <0%> (+3.33%)`	⬆️
src/gluonnlp/data/corpora/wikitext.py	`100% <0%> (+5.17%)`	⬆️
src/gluonnlp/embedding/token_embedding.py	`91.96% <0%> (+5.69%)`	⬆️
... and 21 more

mli · 2019-08-07T18:07:19Z

Job PR-866/1 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-866/1/index.html

mli · 2019-08-08T17:18:34Z

Job PR-866/2 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-866/2/index.html

mli · 2019-08-16T14:19:33Z

Job PR-866/4 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-866/4/index.html

mli · 2019-08-16T17:59:20Z

Job PR-866/5 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-866/5/index.html

eric-haibin-lin · 2019-08-17T00:20:36Z

Is this ready?

leezu · 2019-08-17T07:18:55Z

This only contains the model and no fine-tuning scripts. For finetuning, designing a joint API encompasses the existing bert scripts and works for XLNet and moving that to the main package would be a way forward. That can be done in a separate PR though I guess.

eric-haibin-lin

I don't see an example usage of XLNet with tokenizers and how I can get it with xx.get_model API. Are you going to add it?

szha · 2019-09-02T06:03:05Z

@leezu pinging for an update

szha · 2019-09-11T20:54:25Z

@leezu @eric-haibin-lin gentle ping

leezu · 2019-10-08T00:55:27Z

@eric-haibin-lin an example is included now https://github.com/dmlc/gluon-nlp/pull/866/files#diff-820020a4c66a085eb27014cc377a8658

mli · 2019-10-08T01:28:15Z

Job PR-866/7 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-866/7/index.html

mli · 2019-10-08T02:30:54Z

Job PR-866/8 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-866/8/index.html

mli · 2019-10-08T19:39:41Z

Job PR-866/9 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-866/9/index.html

mli · 2019-10-08T22:00:17Z

Job PR-866/10 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-866/10/index.html

mli · 2019-10-08T23:07:17Z

Job PR-866/11 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-866/11/index.html

Currently unused, but included for future-compatibility of parameter files

We can't create a sentencepiece model from the vocabulary alone. As long as GluonNLP does not reimplement sentencepiece tokenization, the binary model needs to distributed as well.

mli · 2019-10-09T17:25:42Z

Job PR-866/12 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-866/12/index.html

Avoid concurrency failure in gluon's get_model_file

mli · 2019-10-10T18:39:58Z

Job PR-866/13 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-866/13/index.html

mli · 2019-10-10T19:12:53Z

Job PR-866/15 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-866/15/index.html

leezu · 2019-10-10T23:41:07Z

CI passes now

leezu requested a review from szha as a code owner August 7, 2019 17:34

leezu force-pushed the xlnet branch from 3a29a1d to a66cf8b Compare August 8, 2019 16:32

leezu force-pushed the xlnet branch from a66cf8b to c08099f Compare August 16, 2019 13:11

leezu requested review from eric-haibin-lin and sxjscience August 16, 2019 18:04

eric-haibin-lin reviewed Aug 18, 2019

View reviewed changes

leezu requested a review from a team as a code owner September 25, 2019 21:34

leezu requested a review from eric-haibin-lin October 8, 2019 00:54

leezu force-pushed the xlnet branch from 2d43491 to 7d22ebb Compare October 8, 2019 01:56

leezu force-pushed the xlnet branch from f51cc28 to 1e2083e Compare October 8, 2019 21:23

szha added the release focus Progress focus for release label Oct 8, 2019

leezu added 4 commits October 9, 2019 16:49

Add pretrained XLNet

7c7b3ee

Fix lint

e1a895e

Unify TransformerXL and XLNet _rel_shift implementations and APIs

1e60d04

Fix

523e4c0

leezu added 12 commits October 9, 2019 16:49

Update test

67ae610

get_model XLNet

3d38883

Add XLNetTokenizer

9f862f0

Add use_decoder parameter

762916c

Update docstrings

bef88ba

Add mask_embed parameter

26f8ae6

Currently unused, but included for future-compatibility of parameter files

Add test

5776f5d

Distribute sentencepiece model for tokenizer

d3cfbde

We can't create a sentencepiece model from the vocabulary alone. As long as GluonNLP does not reimplement sentencepiece tokenization, the binary model needs to distributed as well.

Rename pytorch_transformers to transformers

c7fd0be

Add examples

7bd0ce6

Fix lint

c5137f1

Update test

efec02a

leezu force-pushed the xlnet branch from e2831f5 to efec02a Compare October 9, 2019 16:49

leezu added 3 commits October 10, 2019 18:02

Print hostname to CI log

25af8f2

Include inode

201e641

Mark test_xlnet_pretrained as serial

d5283b1

Avoid concurrency failure in gluon's get_model_file

szha approved these changes Oct 16, 2019

View reviewed changes

leezu merged commit 2b18e51 into dmlc:master Oct 16, 2019

leezu deleted the xlnet branch October 16, 2019 22:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MODEL] XLNet conversion scripts #866

[MODEL] XLNet conversion scripts #866

leezu commented Aug 7, 2019 •

edited by szha

Loading

codecov bot commented Aug 7, 2019 •

edited

Loading

mli commented Aug 7, 2019

mli commented Aug 8, 2019

mli commented Aug 16, 2019

mli commented Aug 16, 2019

eric-haibin-lin commented Aug 17, 2019

leezu commented Aug 17, 2019

eric-haibin-lin left a comment

szha commented Sep 2, 2019

szha commented Sep 11, 2019

leezu commented Oct 8, 2019

mli commented Oct 8, 2019

mli commented Oct 8, 2019

mli commented Oct 8, 2019

mli commented Oct 8, 2019

mli commented Oct 8, 2019

mli commented Oct 9, 2019

mli commented Oct 10, 2019

mli commented Oct 10, 2019

leezu commented Oct 10, 2019

[MODEL] XLNet conversion scripts #866

[MODEL] XLNet conversion scripts #866

Conversation

leezu commented Aug 7, 2019 • edited by szha Loading

Description

Checklist

Essentials

Changes

Comments

codecov bot commented Aug 7, 2019 • edited Loading

Codecov Report

mli commented Aug 7, 2019

mli commented Aug 8, 2019

mli commented Aug 16, 2019

mli commented Aug 16, 2019

eric-haibin-lin commented Aug 17, 2019

leezu commented Aug 17, 2019

eric-haibin-lin left a comment

Choose a reason for hiding this comment

szha commented Sep 2, 2019

szha commented Sep 11, 2019

leezu commented Oct 8, 2019

mli commented Oct 8, 2019

mli commented Oct 8, 2019

mli commented Oct 8, 2019

mli commented Oct 8, 2019

mli commented Oct 8, 2019

mli commented Oct 9, 2019

mli commented Oct 10, 2019

mli commented Oct 10, 2019

leezu commented Oct 10, 2019

leezu commented Aug 7, 2019 •

edited by szha

Loading

codecov bot commented Aug 7, 2019 •

edited

Loading