Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix null byte \x00 issue in byte level fsm resulting in KeyError in BetterFSM::FSMInfo #930

Merged
merged 8 commits into from
Jun 5, 2024
Merged

Fix null byte \x00 issue in byte level fsm resulting in KeyError in BetterFSM::FSMInfo #930

merged 8 commits into from
Jun 5, 2024

Commits on Jun 4, 2024

  1. Fix null byte \x00 issue by switching to numba.types.unicode_type

    @M0gician @lapp0
    M0gician authored and lapp0 committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    222a5d2 View commit details
    Browse the repository at this point in the history

  2. ensure byte fsm unicode_type compatibility by prefixing hex-bytes wit… 

    …h \x00
    @lapp0
    lapp0 committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    5ba4c2b View commit details
    Browse the repository at this point in the history

  3. index token -> transition key sequence for efficient _walk_fsm

    @lapp0
    lapp0 committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    22d3ddd View commit details
    Browse the repository at this point in the history

  4. pass token_transition_sequence to walk_fsm in parsing.py

    @lapp0
    lapp0 committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    ae9635c View commit details
    Browse the repository at this point in the history

  5. add assertions for numba behavior re: UnicodeCharSeq / unicode_type

    @lapp0
    lapp0 committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    a759728 View commit details
    Browse the repository at this point in the history

  6. Add docstrings to get_token/vocab_transition_keys, improve naming

    @lapp0
    lapp0 committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    1e39027 View commit details
    Browse the repository at this point in the history

  7. remove cleanup step redundant with cleanup_lark_import

    @lapp0
    lapp0 committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    4468180 View commit details
    Browse the repository at this point in the history

  8. Improve naming of token and vocab transition keys in regex.py

    @lapp0
    lapp0 committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    501520a View commit details
    Browse the repository at this point in the history