Make ERB, Handlebars, PHP and Smarty highlight properly in NodeJS

[Golmote]

Since ERB, Handlebars, PHP and Smarty currently rely on before-highlight and after-highlight hooks to be highlighted properly, they can't be easily highlighted using directly the Prism.highlight() function.

#1357 added the after-tokenize hook to post-process tokens directly in the Prism.highlight() function. This PR adds a before-tokenize hook, and essentially translates what we were previously doing with string replacement, but using the array of tokens.

This should remove the need for the backupCode/before-insert hack, since the code is properly tokenized at the end of the highlight() function.

I also converted the tests from .js format to .test format, since the test runner can now run them normally.

[mAAdhaTTah]

Is walkTokens similar enough that we can make that reusable in some way between the languages?

[Golmote]

I guess it could, at least for ERB, Handlebars and Smarty (PHP is slightly different). But where would we put it? I'm not really in favor of adding it directly to prism-core...

[paladox]

Testing this on https://codepen.io/anon/pen/dmzpKg it dosen't seem to work.

[mAAdhaTTah]

I was thinking the Prism.util namespace, but if you don't think it belongs there, it's fine, especially if it's not universally reusable (cuz PHP is different).

[Golmote]

Testing this on https://codepen.io/anon/pen/dmzpKg it dosen't seem to work.

Ruby needs to be included before ERB. See https://codepen.io/anon/pen/eMEgNo

I was thinking the Prism.util namespace, but if you don't think it belongs there, it's fine, especially if it's not universally reusable (cuz PHP is different).

I think this is too much code, used by too few languages (3 or 4 out of 141), to belong in prism-core. But I will keep thinking about this, cause I agree there's far too much repetition here. 😕

[mAAdhaTTah]

Possibly-terrible-idea: Have all of them depend on a "language" who's sole responsibility is to make this function available on Prism.util.

[Golmote]

Maybe this can be something like prism-templating.js since it's their common point: we need this extra replacement process because those are templating languages, that can be embedded everywhere inside Markup.

I'll give it a try this weekend, we'll see how this goes.

(On a side note, Twig and Django components should probably be rewritten in the same fashion, since they belong to the same kind of languages. Although it can be done at anytime later)

[Golmote]

Ok, so I gave this a try and it works really well.

I managed to make all four of the languages behave the same way. Their definition does not extend Markup anymore. Instead, like PHP does, they switch the grammar to Markup in before-tokenize and use their own grammar in after-tokenize. I guess this could even be a step towards a more generic templating behaviour (not focused only on Markup).

Sorry the diff is getting a bit hard to read.

[mAAdhaTTah]

I like this approach, this makes the language implementations a lot cleaner.