Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New tokenize-complete hook #2949

Closed
RunDevelopment opened this issue Jun 13, 2021 · 1 comment
Closed

New tokenize-complete hook #2949

RunDevelopment opened this issue Jun 13, 2021 · 1 comment

Comments

@RunDevelopment
Copy link
Member

Motivation
Some languages have custom tokenizers implemented as hooks (e.g. JSX). Right now, they all use the after-tokenize hook. This is a problem because components like Markup Templating don't run this hook (because they can't, see the implementation).

Therefore, we need a new hook that gets run before after-tokenize and inside the Prism.tokenize function itself.

Description
Since the hook is supposed to be called inside Prism.tokenize, we need to change the function a little. It needs to take a new optional argument: the id of the grammar (the language string).

Only if this argument is present, will the hook be run. This ensures that the hook is only run for top-level grammars and not inside grammars.

Kinda like this:

function tokenize(text: string, grammar: Grammar, language?: string): TokenStream {
	// ...
	matchGrammar(text, tokenList, grammar, tokenList.head, 0);

	var tokens = toArray(tokenList);

	if (language) {
		var env = { tokens, language };
		Prism.hooks.run('tokenize-complete', env);
		return env.tokens;
	} else {
		return tokens;
	}
}

This approach will also make it easy to implement language-specific parsers.


Together with #2948, these languages specific parsers will also work as embedded languages.

@RunDevelopment
Copy link
Member Author

Closed in favor of #3539.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant