You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Motivation
Some languages have custom tokenizers implemented as hooks (e.g. JSX). Right now, they all use the after-tokenize hook. This is a problem because components like Markup Templating don't run this hook (because they can't, see the implementation).
Therefore, we need a new hook that gets run before after-tokenize and inside the Prism.tokenize function itself.
Description
Since the hook is supposed to be called inside Prism.tokenize, we need to change the function a little. It needs to take a new optional argument: the id of the grammar (the language string).
Only if this argument is present, will the hook be run. This ensures that the hook is only run for top-level grammars and not inside grammars.
Kinda like this:
functiontokenize(text: string,grammar: Grammar,language?: string): TokenStream{// ...matchGrammar(text,tokenList,grammar,tokenList.head,0);vartokens=toArray(tokenList);if(language){varenv={ tokens, language };Prism.hooks.run('tokenize-complete',env);returnenv.tokens;}else{returntokens;}}
This approach will also make it easy to implement language-specific parsers.
Together with #2948, these languages specific parsers will also work as embedded languages.
The text was updated successfully, but these errors were encountered:
Motivation
Some languages have custom tokenizers implemented as hooks (e.g. JSX). Right now, they all use the
after-tokenize
hook. This is a problem because components like Markup Templating don't run this hook (because they can't, see the implementation).Therefore, we need a new hook that gets run before
after-tokenize
and inside thePrism.tokenize
function itself.Description
Since the hook is supposed to be called inside
Prism.tokenize
, we need to change the function a little. It needs to take a new optional argument: the id of the grammar (the language string).Only if this argument is present, will the hook be run. This ensures that the hook is only run for top-level grammars and not inside grammars.
Kinda like this:
This approach will also make it easy to implement language-specific parsers.
Together with #2948, these languages specific parsers will also work as embedded languages.
The text was updated successfully, but these errors were encountered: