Parser-driven masking updates for basic "off-line" vocabulary pre-parsing #171
Conversation
brandonwillard
added
text
brandonwillard
force-pushed
the
basic-preparsing-updates
branch
from
14257c7
to
7011735
Compare
| ) | ||
| with pytest.raises(UnexpectedToken): | ||
| parse_to_end(parser_state) | ||
|
|
||
|
|
||
| def test_map_partial_states_to_vocab_regex(): |
There was a problem hiding this comment.
The test test_map_partial_states_to_vocab_regex is a condensed and complete illustration of the partial matching and vocabulary pre-parsing approach. It shows how a partial state index/map is created from a vocabulary and how it can be used to efficiently determine the support of the next-token distribution—without parsing each element of the vocabulary or sampling full sequences from an LM and accepting/rejecting them.
| return False | ||
| return True | ||
|
|
||
| pstate_to_vocab = map_partial_states_to_vocab( |
There was a problem hiding this comment.
We should add function that maps the states of masks over the vocabulary, so these are not re-computed at each step (provided the memory footprint is not too large). But it can be done in a later PR when needed, #166 for instance.
|
|
||
|
|
||
| def test_map_partial_states_to_vocab_regex(): | ||
| regex_string = r"(([0-9]+)?([.]([0-9]*)?)?|[.][0-9]+)" |
There was a problem hiding this comment.
More of a reminder for myself later: this regex matches floats with leading zeros e.g. "01.34". The following regex forbids them: ^((0|[1-9][0-9]+)?([.]([0-9]*)?)?|[.][0-9]+)$
|
I only had a couple comments that don't necessarily need to be addressed here. LGTM. |
This PR picks up from #131 by extending/refactoring the partial parsing functionality so that it can be used more easily to pre-"parse" vocabularies in cases simpler than general context-free grammars (e.g. regex-only masking).
This is a subset of the functionality implied by https://github.com/normal-computing/outlines/issues/170.