v0.1.97

Major changes

• Migrated the C++ version from C++11 to C++17.
• Migrated the CI environment from Travis-CI to Github actions
• Started using cibuildtool to build pypi wheel packages

New features

• [ALL] Support differential privacy while training. https://aclanthology.org/2022.findings-acl.171.pdf
• [ALL] Introduced APIs that return the struct of ImmutableSentencePieceText, which encodes string-token, id, and utf-8 byte offsets at once. New API is available both from C++ and Python.
• [ALL] Allow tab ‘\t’ to be included in user defined symbols.
• [ALL] Added NFKD normalization rule. NFKD rule is provided as a TSV file.
• [ALL] Added option to emit unknown symbol instead of raw symbol.
• [Python]: Batch encode/decode requests are performed in native multi-threads.
• [Python]: Supports to pass a custom log stream during training.
• [Python]: Adds module-level version variable: spm.__version__
• [Python]: Creates wheel package of Mac universal binary.

Bug fixes & minor changes

• Uses the efficient encoding algorithm by default. Removed the functionality to switch the Viterbi tokenization algorithm.
• Make the output of Encode and 1-best from NBestEncode same.
• Use std::string_view as much as possible.
• [Python] Removed pip package for ppc64le and s390x architecture as cibuiltool doesn’t support them.