This is a series of scripts dedicated to analyzing Internet comments about an undisclosed Canadian company.
- Run lda.py on a .csv/.tsv/etc. file. The input table must be in the following format:
| insert_date | text | language | ... |
|---|---|---|---|
| 2019-05-27 21:06:48 | keep up the good work | en | ... |
| 2019-05-27 21:06:48 | Vendez un Bar-B-Q déjà toute monté | fr | ... |
| ... | ... | ... | ... |
- Afterwards, you can use
- keyword_analysis.py: corpus comparison based on the K-means algorithm, keyword search among comments using logical expressions or SpaCy rule-based matching;
- similarity_model.py: word2vec model training on the obtained corpus and the search among keywords that is based on it.
New entities for entity_recognition.py script can be added here. This file uses SpaCy pattern keys. For more information, see available token attributes and available labels.
Unicode symbols (such as è) must be written using their respective source code (in this case, \u00e8). Don't put any whitespaces, as they'll break the script.
Finally, if you wish to exclude a specific entity from SpaCy lemmatization (for example, you want "food_basics" to always stay plural), add "/l-excluded" to its id (see the file itself for some examples).
This dictionary is based on a British-American one from this site, and, though not comprehensive, it gets the job done.
New entries can be added in the non-canadian-spelling'\t'canadian-spelling format (see the file).