Add LayoutLM Model

[liminghao1630]

Introduction

This pull request implements the LayoutLM model, as defined in the paper:

LayoutLM: Pre-training of Text and Layout for Document Image Understanding 
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou, KDD 2020

LayoutLM is a simple but effective pre-training method of text and layout for document image understanding and information extraction tasks, such as form understanding and receipt understanding. LayoutLM archives the SOTA results on multiple datasets.

Typical workflow for including a model

Here an overview of the general workflow:

• Add model/configuration/tokenization classes.
• Add conversion scripts.
• Add tests and a @slow integration test.
• Document your model.
• Finalize.

Let's detail what should be done at each step.

Adding model/configuration/tokenization classes

Here is the workflow for adding model/configuration/tokenization classes:

• Copy the python files from the present folder to the main folder and rename them, replacing xxx with your model
  name.
• Edit the files to replace XXX (with various casing) with your model name.
• Copy-paste or create a simple configuration class for your model in the configuration_... file.
• Copy-paste or create the code for your model in the modeling_... files (PyTorch and TF 2.0).
• Copy-paste or create a tokenizer class for your model in the tokenization_... file.

Adding conversion scripts

Here is the workflow for the conversion scripts:

• Copy the conversion script (convert_...) from the present folder to the main folder.
• Edit this script to convert your original checkpoint weights to the current pytorch ones.

Adding tests:

Here is the workflow for the adding tests:

• Copy the python files from the tests sub-folder of the present folder to the tests subfolder of the main
  folder and rename them, replacing xxx with your model name.
• Edit the tests files to replace XXX (with various casing) with your model name.
• Edit the tests code as needed.

Documenting your model:

Here is the workflow for documentation:

• Make sure all your arguments are properly documented in your configuration and tokenizer.
• Most of the documentation of the models is automatically generated, you just have to make sure that
  XXX_START_DOCSTRING contains an introduction to the model you're adding and a link to the original
  article and that XXX_INPUTS_DOCSTRING contains all the inputs of your model.
• Create a new page xxx.rst in the folder docs/source/model_doc and add this file in docs/source/index.rst.

Make sure to check you have no sphinx warnings when building the documentation locally and follow our
documentaiton guide.

Final steps

You can then finish the addition step by adding imports for your classes in the common files:

• Add import for all the relevant classes in __init__.py.
• Add your configuration in configuration_auto.py.
• Add your PyTorch and TF 2.0 model respectively in modeling_auto.py and modeling_tf_auto.py.
• Add your tokenizer in tokenization_auto.py.
• Add a link to your conversion script in the main conversion utility (in commands/convert.py)
• Edit the PyTorch to TF 2.0 conversion script to add your model in the convert_pytorch_checkpoint_to_tf2.py
  file.
• Add a mention of your model in the doc: README.md and the documentation itself
  in docs/source/index.rst and docs/source/pretrained_models.rst.
• Upload the pretrained weights, configurations and vocabulary files.
• Create model card(s) for your models on huggingface.co. For those last two steps, check the
  model sharing documentation.

[codecov]

Codecov Report

Merging #7064 into master will decrease coverage by 0.09%.
The diff coverage is 31.25%.

@@            Coverage Diff             @@
##           master    #7064      +/-   ##
==========================================
- Coverage   80.32%   80.23%   -0.10%     
==========================================
  Files         174      177       +3     
  Lines       33446    33878     +432     
==========================================
+ Hits        26867    27181     +314     
- Misses       6579     6697     +118     

Impacted Files	Coverage Δ
src/transformers/modeling_layoutlm.py	25.56% <25.56%> (ø)
src/transformers/activations.py	86.36% <50.00%> (-3.64%)	⬇️
src/transformers/configuration_layoutlm.py	80.00% <80.00%> (ø)
src/transformers/__init__.py	99.37% <100.00%> (+0.01%)	⬆️
src/transformers/configuration_auto.py	96.20% <100.00%> (+0.04%)	⬆️
src/transformers/modeling_auto.py	82.46% <100.00%> (+0.08%)	⬆️
src/transformers/tokenization_auto.py	92.30% <100.00%> (+0.12%)	⬆️
src/transformers/tokenization_layoutlm.py	100.00% <100.00%> (ø)
src/transformers/modeling_tf_funnel.py	18.53% <0.00%> (-75.51%)	⬇️
src/transformers/modeling_tf_flaubert.py	24.53% <0.00%> (-63.81%)	⬇️
... and 18 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7cbf0f7...4e95220. Read the comment docs.

[JetRunner]

The PR is overall in a good shape! Thanks for your work!

@sgugger @patrickvonplaten Interested in this?

[JetRunner]

I think you may find torch.view() better here.

[JetRunner]

same here

[sgugger]

Thanks a lot for adding this model! Before we can merge, there is a tiny bit of work to do to make the modeling file independent from bert_modeling.py. Could you copy the classes you use from that file to the layeroutlm_modeling file (and rename them from BertXxx to LayeroutLMXxx), you can look at the files for GPT and GPT-2 as example. The idea is that this way, any researcher can directly tweak the model by copying its file without having to worry about extra stuff.

Otherwise, all look good to me apart form a few nits.

[liminghao1630]

Hi, I have moved and renamed the classes from modeling_bert to modeling_layoutlm and the code is ready for review and merge. @JetRunner @sgugger

[sgugger]

Thanks for the copied code. This all looks good to me and ready to merge, with just one nit.

[LysandreJik]

This is exceptionally clean! Great job!

[liminghao1630]

@JetRunner @sgugger @LysandreJik The new suggestions have been adopted.

[LysandreJik]

Great, thanks a lot @liminghao1630