Highlight JS (tagged) template literals

[RunDevelopment]

This PR adds support for arbitrary syntax highlighting within JS template literals via a new JS Templates language/extension.

Currently implemented are most ideas from #1930:

• Markup
  • ✅ html`<p></p>`
  • ✅ div.innerHTML = `<p></p>`
• CSS
  • ✅ css`a { color: red; }`
  • ✅ styled.foo`color: red;`
  • ⛔ Component.extend`color: red;` Since this is deprecated, I don't think we have to support it.
  • ❓ <style jsx>{`div{color:red}`}</style>
    Implementing this will likely be a bit harder because of the way JSX matches text. But this implementation won't be part of JS Templates, so it's not included in this PR.
• Markdown
  • ✅ md`# title`
• GraphQL
  • ✅ gql`{ foo }`
• Other
  • ⛔ Angular Components HTML + CSS:

    @Component({
      template: `<div>...</div>`,
      styles: [`h1 { color: blue; }`]
    })
    

    Implementing this in regexes is next to impossible. Prettier implemented this using the created AST which is something Prism doesn't have.
  • ⛔ /* HTML */`...` Not included because I don't want to introduce Prsim-specific syntax for the purpose of highlighting.

Example

---

This is based on #1714.

[luwes]

Will be awesome to have this in Prism. Nice work!

Could the svg tag possibly be also added?

createTemplate('svg', /\bsvg|\.\s*(?:inner|outer)HTML\s*=/.source, 'markup'),

Quite a few libs have this tag separately from the html tag.

[luwes]

Thanks @RunDevelopment! I already put your code to use, https://sinuous.netlify.com/examples/clock/

There seems to be one issue with the html/svg tags that have an expression in them, they have other colors.

[luwes]

Found a fix I think. The issue was that the markup pattern regex was too strict w/ the included interpolations. The following is a snippet of the working code.

If it's markup, take the extended lang template-string-markup.
Here's the diff of the regex.

    const lang = language === 'markup' ? 'template-string-markup' : language;
    patternObj.inside[tokenName] = !language ? /[\s\S]+/ : {
      pattern: /[\s\S]+/,
      alias: 'language-' + language,
      inside: Prism.languages[lang]
    };

    return patternObj;
  }

});

const templateMarkup = Prism.languages['template-string-markup'] = Prism.languages.extend('markup', {});

templateMarkup.tag.pattern = /<\/?(?!\d)[^\s>\/=$<%]+(?:\s(?:\s*[^\s>=]+(?:\s*=\s*(?:"[^"]*"|'[^']*'|[^\s'">=]+(?=[\s>])|$)|(?=[\s/>])))+)?\s*\/?/i;

Prism.languages.javascript['template-string'].push(Prism.languages.javascript['template-string'].createTemplate('string'));

[RunDevelopment]

The solution is really not optimal. Instead of relaxing the HTML tag pattern we should do something similar to Markup templating (MT).
MT first searches for all parts matching a pattern (in our case that the string interpolation) and cuts them out (put simply). Then MT highlights the markup and inserts the cut-out pieces again. This approach is very reliable and works great for embedded languages (e.g. PHP, EJS, ...) but my problem is that it only does this for Markup languages.

I also want to do the same for all other embedded languages, so I could either adjust MT to support more than just Markup (which kind of invalidates the name) or I could write my own MT without the Markup focus.
I'm currently contemplating as to what I'll do but I'm leaning towards the first option.

---

As to why relaxing the pattern isn't the best solution: A relaxed pattern will also match syntactically invalid code and are likely to break in more complex situations. Also, you relaxed just one pattern when you actually had to relax ALL patterns to make the string interpolation for other tokens as well. This is a whole lot of work and had to be done for every embedded language.

[luwes]

Thanks for the reply, that makes a lot of sense. I'll have a look at MT.
I also just noticed my solution was breaking for markup after the interpolations :s.

[RunDevelopment]

I'll have a look at MT

Don't worry. I'll do it.
I might as well fix the bugs of my own PR 😉

[mAAdhaTTah]

Looking over the PRs, one thing that occurs to me is we've got a few implementations of "highlighting one language in another." I'm going to take another pass over them individually, but I wanted to ask: is there any logic that could be shared between them, a la markup-templating?

[RunDevelopment]

We've got a few implementations of "highlighting one language in another."

Have we? I only know all the languages which use Markup templating and the ones which embed languages literally like HTML embeds CSS & JS.
The excessive amount of code I wrote is only necessary because of the string interpolations which require tokenizing the code in multiple passes.

Even though I'll admit that I'm guilty of copying a few lines from MT, I don't think that there is a whole lot to share. At least, not now.

[mAAdhaTTah]

We have markdown, js template strings, & diff, which do highlighting of one language in another. They don't have to use markup-templating, but are there enough similarities that they can use shared logic. Sounds like no, that's fine, just wanted to put the question out there.

[mAAdhaTTah]

What if we made the dependencies peerDependencies so you wouldn't need to include graphql if you aren't using it?

[RunDevelopment]

Sounds like no

Sorry, that's not what I wanted to say. Diff, JS Templating, and Markup Templating don't share a lot of logic because they have different solutions for the same problem.

I absolutely think that we could make a more general templating engine which would be able to make the above three obsolete. I actually tried just that in the process of making this but MT is very different from the solution I employed here. I would most likely have to adjust every language using MT, so I didn't want to include this here.
That being said, I do think that a more general templating engine would be a great addition to Prism.

[RunDevelopment]

Regarding the peer dependencies:

Should the templating be optional as well, so we don't even add the template if the embedded language isn't loaded?

RN, missing grammar look like this.

[mAAdhaTTah]

Should the templating be optional as well, so we don't even add the template if the embedded language isn't loaded?

Yeah, that sounds good to me.

[RunDevelopment]

I want to cry a little: The test suite can't handle peer dependencies.
I hope modifying the path works...

[mAAdhaTTah]

Don't we have a syntax to load languages in the tests? We've got these folders like django!+css, which I believe loads both those languages.

[RunDevelopment]

Yup, that's what I meant with "modifying the path".
The solution is actually quite simple now that I look at it more closely.

Missing changes? walkTokens is still duped.

[RunDevelopment]

I renamed one of the functions. Wasn't that what you meant?
Commented as your comment loaded.

[mAAdhaTTah]

Yeah, you renamed the second function, which I missed when I was looking over the code. My b.