Skip to content

p16i/Syllable-based-Neural-Thai-Word-Segmentation

Repository files navigation

Syllable based Neural Thai Word-Segmentation

By Pattarawat Chormai, Ponrawee Prasertsom, Jin Cheevaprawatdomrong, and Attapol T. Rutherford

Links: [Paper 📑] [Poster 🖼] [Citation ⚓️️]

Related Works:

🚧 for running the code, please consult DEV.md. The code was developed using Python 3.8.7.

Highlights

Syllable and Word Segmentation Performance

Expected Validation Performance

Explaining Word Segmentation

Model Statistics

Statistic Files
Model Statistics File
BiLSTM(CH)-BI seq_ch_lstm_bi.yaml-2020-06-04--09-17.20.csv
BiLSTM(CH-SY)-BI seq_sy_ch_lstm_bi.yaml-2020-06-03--20-26.20.csv
BiLSTM(SY)-SchemeBI seq_sy_lstm_bi.yaml-2020-06-03--23-35.20.csv
BiLSTM(SY)-SchemeA seq_sy_lstm_scheme_a.yaml-2020-06-03--23-35.20.csv
BiLSTM(SY)-SchemeB seq_sy_lstm_scheme_b.yaml-2020-06-03--23-35.20.csv
BiLSTM-CRF(SY)-BI seq_sy_lstm_bi_crf.yaml-2020-06-03--18-10.20.csv
BiLSTM-CRF(SY)-SchemeA seq_sy_lstm_crf_scheme_a.yaml-2020-06-03--23-34.20.csv
BiLSTM-CRF(SY)-SchemeB seq_sy_lstm_crf_scheme_b.yaml-2020-06-03--23-35.20.csv
ID-CNN(CH)-BI seq_ch_conv_3lv.yaml-2020-06-03--12-11.20.csv
ID-CNN(CH-SY)-BI seq_sy_ch_conv_3lv.yaml-2020-06-02--23-23.20.csv
ID-CNN(SY)-BI seq_sy_conv_3lv.yaml-2020-06-02--08-19.20.csv
ID-CNN(SY)-SchemeA seq_sy_conv_3lv_scheme_a.yaml-2020-06-02--10-49.20.csv
ID-CNN(SY)-SchemeB seq_sy_conv_3lv_scheme_b.yaml-2020-06-02--10-49.20.csv
ID-CNN-CRF(SY)-BI seq_sy_conv_3lv_crf_bi.yaml-2020-06-01--11-40.20.csv
ID-CNN-CRF(SY)-SchemeA seq_sy_conv_3lv_crf_scheme_a.yaml-2020-06-01--11-39.20.csv
ID-CNN-CRF(SY)-SchemeB seq_sy_conv_3lv_crf_scheme_b.yaml-2020-06-01--11-39.20.csv

Citation

@inproceedings{chormai-etal-2020-syllable,
    title = "Syllable-based Neural {T}hai Word Segmentation",
    author = "Chormai, Pattarawat  and
      Prasertsom, Ponrawee  and
      Cheevaprawatdomrong, Jin  and
      Rutherford, Attapol",
    booktitle = "Proceedings of the 28th International Conference on Computational Linguistics",
    month = dec,
    year = "2020",
    address = "Barcelona, Spain (Online)",
    publisher = "International Committee on Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.coling-main.407",
    pages = "4619--4637",
    abstract = "Word segmentation is a challenging pre-processing step for Thai Natural Language Processing due to the lack of explicit word boundaries.The previous systems rely on powerful neural network architecture alone and ignore linguistic substructures of Thai words. We utilize the linguistic observation that Thai strings can be segmented into syllables, which should narrow down the search space for the word boundaries and provide helpful features. Here, we propose a neural Thai Word Segmenter that uses syllable embeddings to capture linguistic constraints and uses dilated CNN filters to capture the environment of each character. Within this goal, we develop the first ML-based Thai orthographical syllable segmenter, which yields syllable embeddings to be used as features by the word segmenter. Our word segmentation system outperforms the previous state-of-the-art system in both speed and accuracy on both in-domain and out-domain datasets.",
}

About

No description or website provided.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published