BLU is a chatbot that generates text in Polish based on the Transformer model and trained using data from movie subtitles collected through web scraping.
Polish is one of the most complicated languages in the world. Hence, there are not many training datasets for language models. Therefore, it was decided to create such a dataset, from files with movie subtitles in Polish, which were collected using the BeautifulSoup script. Conversation pairs were created from SRT files.
| Version | Number of conversation pairs |
|---|---|
| I | 197 212 |
| II | 283 389 |
| III | 1 008 839 (*) |
(*) In version III, some conversation pairs are duplicated.
Based on the self-created language corpus, several versions of models were trained, which were later used to build a chatbot.
The Transformer model was used, its implementation is based on tutorials linked in the bibliography.
Different versions of hyperparameters were tested and finally 3 versions of models were trained - based on 3 versions of data sets.
For logical correctness testing, 384 different messages were prepared and delivered to the chatbot with different versions of the model loaded. The test set included both questions and declarative sentences. The topics of the test set were diverse and included a range of statements that could be used during a classic, everyday conversation.
Results of testing the logical correctness of the BLU response:
| Model | Number of correct, logical answers | Level of logical correctness |
|---|---|---|
| I | 156 | 40,63% |
| II | 158 | 41,15% |
| III | 187 | 48,7% |