3.1.29

[3.1.29]

Changed

• Running sockeye-evaluate no longer applies text tokenization for TER (same behavior as other metrics).
• Turned on type checking for all sockeye modules except test_utils and addressed resulting type issues.
• Refactored code in various modules without changing user-level behavior.

[3.1.28]

Added

• Added kNN-MT model from Khandelwal et al., 2021.
  • Installation: see faiss document -- installation via conda is recommended.
  • Building a faiss index from a sockeye model takes two steps:
    • Generate decoder states: sockeye-generate-decoder-states -m [model] --source [src] --target [tgt] --output-dir [output dir]
    • Build index: sockeye-knn -i [input_dir] -o [output_dir] -t [faiss_index_signature] where input_dir is the same as output_dir from the sockeye-generate-decoder-states command.
    • Faiss index signature reference: see here
  • Running inference using the built index: sockeye-translate ... --knn-index [index_dir] --knn-lambda [interpolation_weight] where index_dir is the same as output_dir from the sockeye-knn command.