You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit brings a BIG update to the tokenization algorithm: instead of
creating "recursive" tokens, the complete tokenization of all of the source
text now happens iteratively in a single loop. The nifty `--debug` command
line option lets you inspect how the input is tokenized.
The new algorithm reduces the complexity of the modes to just a single
`create_token()` method, solves the line numbering problem (#18), and solves
various incorrect tokenization issues (such as #20).
I would like to thank my former professor J.J. Paijmans for teaching me the
knowledge required to pull of this feat. Thanks Paai!
Besides this, a new mode `InsideHTMLTag` was added to allow for different
rules within HTML tags than outside of it. This means that multiline html
tags are now tokenized and parsed correctly.
Finally, the list of Django block tags is gone and they are now recognized
by looking ahead for end tags, which eliminates the need for a configuration
option to register custom template tags (#16).
Closes#16Closes#18 (again)
Closes#20
Using the 4-line excerpt
the output is
but should have been
The text was updated successfully, but these errors were encountered: