Skip to content
This repository has been archived by the owner on Apr 23, 2024. It is now read-only.

[WIP] fast wordpiece tokenization #105

Open
wants to merge 11 commits into
base: master
Choose a base branch
from
Open

[WIP] fast wordpiece tokenization #105

wants to merge 11 commits into from

Commits on Mar 20, 2023

  1. draft wordpiece

    gleb authored and gleb committed Mar 20, 2023
    Configuration menu
    Copy the full SHA
    5b80521 View commit details
    Browse the repository at this point in the history

Commits on Apr 1, 2023

  1. Merge branch 'master' into fast-wordpiece

    @gleb-kov
    gleb-kov committed Apr 1, 2023
    Configuration menu
    Copy the full SHA
    44ac134 View commit details
    Browse the repository at this point in the history

  2. cleanup

    @gleb-kov
    gleb-kov committed Apr 1, 2023
    Configuration menu
    Copy the full SHA
    fd13989 View commit details
    Browse the repository at this point in the history

  3. cleanup 2

    @gleb-kov
    gleb-kov committed Apr 1, 2023
    Configuration menu
    Copy the full SHA
    8e9df07 View commit details
    Browse the repository at this point in the history

  4. format

    @gleb-kov
    gleb-kov committed Apr 1, 2023
    Configuration menu
    Copy the full SHA
    f0ba916 View commit details
    Browse the repository at this point in the history

  5. draft python interface

    @gleb-kov
    gleb-kov committed Apr 1, 2023
    Configuration menu
    Copy the full SHA
    49a1dfe View commit details
    Browse the repository at this point in the history

Commits on Apr 2, 2023

  1. better python interface

    @gleb-kov
    gleb-kov committed Apr 2, 2023
    Configuration menu
    Copy the full SHA
    bb0e275 View commit details
    Browse the repository at this point in the history

  2. binding works

    @gleb-kov
    gleb-kov committed Apr 2, 2023
    Configuration menu
    Copy the full SHA
    506fa94 View commit details
    Browse the repository at this point in the history

Commits on Apr 8, 2023

  1. encoder class, stress tests

    @gleb-kov
    gleb-kov committed Apr 8, 2023
    Configuration menu
    Copy the full SHA
    d7741ec View commit details
    Browse the repository at this point in the history

  2. format

    @gleb-kov
    gleb-kov committed Apr 8, 2023
    Configuration menu
    Copy the full SHA
    4228b28 View commit details
    Browse the repository at this point in the history

Commits on Apr 10, 2023

  1. fix build

    @gleb-kov
    gleb-kov committed Apr 10, 2023
    Configuration menu
    Copy the full SHA
    aa60873 View commit details
    Browse the repository at this point in the history