You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
LexerSnapshot existed due to old reasons which no longer apply. It doesn't have any real reason to exist anymore.
Describe the solution you'd like
Update the parser to take a lexer state and read tokens directly from there.
Describe alternatives you've considered
N/A
Additional context
N/A
The text was updated successfully, but these errors were encountered:
Was lexer snapshot added to support incremental lexing? or can we still do that with state? I'm still hoping we'll eventually be able to support parser based tokenization + semantic highlighting.
Originally it was added to help support it, but it no longer is needed.
The lexer creates tokens on a per-line basis, which requires some additional token kinds such as MultilineCommentStart and MultilineCommentEnd, Whenever one line gets updated it conditionally updates the subsequent lines. Eg. you started a multiline comment on one line then it turns the subsequent line into a MultilineCommentContent.
When you want to actually parse something you need try creating a LexerSnapshot. The snapshot attempt iterates over all of the tokens which provides:
Validate and combine multiline tokens into a single token
Put all comments into a collection
Provides a helper function to get a [lineNumber, columnNumber] pair from a Token
All of these could be moved into the parser and LexerSnapshot could be removed. A trade happens by slightly adding to the complexity of the parser, but also removes complexity of having the LexerSnapshot at all. It also removes an O(n) pass on tokens.
Is your feature request related to a problem? Please describe.
LexerSnapshot existed due to old reasons which no longer apply. It doesn't have any real reason to exist anymore.
Describe the solution you'd like
Update the parser to take a lexer state and read tokens directly from there.
Describe alternatives you've considered
N/A
Additional context
N/A
The text was updated successfully, but these errors were encountered: