[SCRIPT] - Add static BERT base export script (for use with MXNet Module API)

[gigasquid]

Description

This will export the base BERT model for use with the MXNet Module API.

It was adapted from the static_export_squad.py

Use cases can include fine tuning for Clojure and Scala APIs.

• Have used it successfully to reproduce https://gluon-nlp.mxnet.io/examples/sentence_embedding/bert.html in Clojure - PR in MXNet will be coming shortly

Checklist

Essentials

• PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage
• Code is well-documented

Comments

Python is not my usual language - so feedback is welcome :)

[codecov]

Codecov Report

Merging #672 into master will increase coverage by 0.1%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##           master     #672     +/-   ##
=========================================
+ Coverage   90.94%   91.04%   +0.1%     
=========================================
  Files          64       64             
  Lines        5887     5887             
=========================================
+ Hits         5354     5360      +6     
+ Misses        533      527      -6

Impacted Files	Coverage Δ
src/gluonnlp/data/dataloader.py	88.79% <0%> (+5.17%)	⬆️

[gigasquid]

Nice bot 🤖

[mli]

Job PR-672/1 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-672/1/index.html

[szha]

Thanks for the contribution, @gigasquid (and glad to see you here!). Do you intend to use such model for feature extraction with mxnet module?

[gigasquid]

@szha The intent is to use of BERT to more than just inference in the Clojure/Scala MXNet apis. Initially, I'm interested in the fine tuning tasks like classification, but feature extraction would be cool too. Basically everything in this http://jalammar.github.io/illustrated-bert/

Thanks so much for the static export feature! It opens up a whole new BERT world for the JVM MXNet langs 💯

@haven-jeon - Good point on the parameters. I'll double check to see if they are all used and remove the ones that are not.

[gigasquid]

I was able to put together a walkthrough Clojure jupyter notebook and then export it to markdown 😸
https://github.com/gigasquid/incubator-mxnet/blob/new-bert-example-with-finetuning/contrib/clojure-package/examples/bert/fine-tune-bert.md

The PR for MXNet is here apache/mxnet#14769

I'll plan on spending some time tomorrow to double check this script and the params.

[gigasquid]

@haven-jeon - I cleaned up some unneeded args and checked the rest.
Tested with python static_export_base.py --seq_length 128

Please take another look when you have a chance.

[mli]

Job PR-672/5 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-672/5/index.html

[szha]

@gigasquid we recently upgraded the CI setup. It will work once you rebase the PR to latest master.

[mli]

Job PR-672/1 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-672/1/index.html

[mli]

Job PR-672/3 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-672/3/index.html

[mli]

Job PR-672/4 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-672/4/index.html

[mli]

Job PR-672/5 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-672/5/index.html

[mli]

Job PR-672/7 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-672/7/index.html

[mli]

Job PR-672/8 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-672/8/index.html

[mli]

Job PR-672/10 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-672/10/index.html

[gigasquid]

tests & docs adds and CI is green 💚 - please take another look when you get a chance

[eric-haibin-lin]

nice work!