Skip to content

Fast + Non-Autoregressive Grammatical Error Correction using BERT. Code and Pre-trained models for paper "Parallel Iterative Edit Models for Local Sequence Transduction": www.aclweb.org/anthology/D19-1435.pdf (EMNLP-IJCNLP 2019)

License

Notifications You must be signed in to change notification settings

awasthiabhijeet/PIE

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

25 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

PIE: Parallel Iterative Edit Models for Local Sequence Transduction

Fast Grammatical Error Correction using BERT

Code and Pre-trained models accompanying our paper "Parallel Iterative Edit Models for Local Sequence Transduction" (EMNLP-IJCNLP 2019)

PIE is a BERT based architecture for local sequence transduction tasks like Grammatical Error Correction. Unlike the standard approach of modeling GEC as a task of translation from "incorrect" to "correct" language, we pose GEC as local sequence editing task. We further reduce local sequence editing problem to a sequence labeling setup where we utilize BERT to non-autoregressively label input tokens with edits. We rewire the BERT architecture (without retraining) specifically for the task of sequence editing. We find that PIE models for GEC are 5 to 15 times faster than existing state of the art architectures and still maintain a competitive accuracy. For more details please check out our EMNLP-IJCNLP 2019 paper

@inproceedings{awasthi-etal-2019-parallel,
    title = "Parallel Iterative Edit Models for Local Sequence Transduction",
    author = "Awasthi, Abhijeet  and
      Sarawagi, Sunita  and
      Goyal, Rasna  and
      Ghosh, Sabyasachi  and
      Piratla, Vihari",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)",
    month = nov,
    year = "2019",
    address = "Hong Kong, China",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/D19-1435",
    doi = "10.18653/v1/D19-1435",
    pages = "4259--4269",
}

Datasets

  • All the public GEC datasets used in the paper can be obtained from here
  • Synthetically created datasets (perturbed version of 1 billion word corpus) divided into 5 parts to independently train 5 different ensembles. (all the ensembles are further finetuned using the public GEC datasets mentioned above.)

Pretrained Models

  • PIE as reported in the paper
    • trained on a Synethically created GEC dataset starting with BERT's initialization
    • finetuned further on Lang8, NUCLE and FCE datasets
  • Inference using the pretrained PIE ckpt
    • Copy the pretrained checkpoint files provided above to PIE_ckpt directory
    • Your PIE_ckpt directory should contain
      • bert_config.json
      • multi_round_infer.sh
      • pie_infer.sh
      • pie_model.ckpt.data-00000-of-00001
      • pie_model.ckpt.index
      • pie_model.ckpt.meta
      • vocab.txt
    • Run: $ ./multi_round_infer.sh from PIE_ckpt directory
    • NOTE: If you are using cloud-TPUs for inference, move the PIE_ckpt directory to the cloud bucket and change the paths in pie_infer.sh and multi_round_infer.sh accordingly

Code Description

An example usage of code in described in the directory "example_scripts".

  • preprocess.sh
    • Extracts common insertions from a sample training data in the "scratch" directory
    • converts the training data in the form of incorrect tokens and aligned edits
  • pie_train.sh
    • trains a pie model using the converted training data
  • multi_round_infer.sh
    • uses a trained PIE model to obtain edits for incorrect sentences
    • does 4 rounds of iterative editing
    • uses conll-14 test sentences
  • m2_eval.sh
    • evaluates the final output using m2scorer
  • end_to_end.sh
    • describes the use of pre-processing, training, inference and evaluation scripts end to end.
  • More information in README.md inside "example_scripts"

Pre processing and Edits related

  • seq2edits_utils.py
    • contains implementation of edit-distance algorithm.
    • cost for substitution modified as per section A.1 in the paper.
    • Adapted from belambert's implimentation
  • get_edit_vocab.py : Extracts common insertions (\Sigma_a set as described in paper) from a parallel corpus
  • get_seq2edits.py : Extracts edits aligned to input tokens
  • tokenize_input.py : tokenize a file containing sentences. token_ids obtained go as input to the model.
  • opcodes.py : A class where members are all possible edit operations
  • transform_suffixes.py: Contains logic for suffix transformations
  • tokenization.py : Similar to BERT's implementation, with some GEC specific changes

PIE model (uses implementation of BERT of bert in Tensorflow)

  • word_edit_model.py: Implementation of PIE for learning from a parallel corpous of incorrect tokens and aligned edits.
    • logit factorization logic (Keep flag use_bert_more=True to enable logit factorization)
    • parallel sequence labeling
  • modeling.py : Same as in BERT's implementation
  • modified_modeling.py
    • Rewires attention mask to obtain representations of candidate appends and replacements
    • Used for logit factorization.
  • optimization.py : Same as in BERT's implementation

Post processing

  • apply_opcode.py
    • Applies inferred edits from the PIE model to the incorrect sentences.
    • Handles punctuations and spacings as per requirements of a standard dataset (INFER_MODE).
    • Contains some obvious rules for captialization etc.

Creating synthetic GEC dataset

  • errorify directory contains the scripts we used for perturbing the one-billion-word corpus

Acknowledgements

This research was partly sponsored by a Google India AI/ML Research Award and Google PhD Fellowship in Machine Learning. We gratefully acknowledge Google's TFRC program for providing us Cloud-TPUs. Thanks to Varun Patil for helping us improve the speed of pre-processing and synthetic-data generation pipelines.

About

Fast + Non-Autoregressive Grammatical Error Correction using BERT. Code and Pre-trained models for paper "Parallel Iterative Edit Models for Local Sequence Transduction": www.aclweb.org/anthology/D19-1435.pdf (EMNLP-IJCNLP 2019)

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published