[Feature Request] official guide for wiring TextMate grammars

[souporserious]

Context

• This issue is not a bug report. (please use a different template for reporting a bug)
• This issue is not a duplicate of an existing issue. (please use the search to find existing issues)

Description

Hello, I've been struggling to find a solution that works with TextMate grammars and the latest releases for monaco-editor and monaco-editor-webpack-plugin. I've tried following the linked monaco-tm package and the issue here #1915, but it seems some of the theme helpers generateTokensCSSForColorMap, TokenizationRegistry, and Color are no longer available. I even read the wonderful blog post about the deep dive into how it's all working to try and better diagnose the issue and just can't seem to get any solution to work.

I previously had this working using monaco-editor-textmate, but that has recently broke between monaco-editor versions 0.29.1 and 0.30.0 zikaari/monaco-editor-textmate#26.

I've managed to distill the wiring of textmate down to the solution below based on monaco-editor-textmate and theia. I know it's not the most performant route per se, but I was hoping to use monaco as an enhancement on top of server-rendered shiki code blocks so this will be loaded on demand if someone wants to play around with code examples.

import type { Registry, StateStack } from 'vscode-textmate'
import { INITIAL } from 'vscode-textmate'
import * as monaco from 'monaco-editor'

/** Wires up monaco-editor with monaco-textmate */
export function wireTextMateGrammars(
  /** TmGrammar `Registry` this wiring should rely on to provide the grammars. */
  registry: Registry,

  /** `Map` of language ids (string) to TM names (string). */
  languages: Map<string, string>,

  /** The monaco editor instance to wire up. */
  editor: monaco.editor.ICodeEditor
) {
  const tokenTheme = editor['_themeService'].getColorTheme().tokenTheme
  const defaultForeground = tokenTheme._root._mainRule._foreground

  return Promise.all(
    Array.from(languages.keys()).map(async (languageId) => {
      const grammar = await registry.loadGrammar(languages.get(languageId))

      monaco.languages.setTokensProvider(languageId, {
        getInitialState: () => new TokenizerState(INITIAL),
        tokenize: (line: string, state: TokenizerState) => {
          const result = grammar.tokenizeLine(line, state.ruleStack)

          return {
            endState: new TokenizerState(result.ruleStack),
            tokens: result.tokens.map((token) => {
              const scopes = token.scopes.slice(0)

              for (let i = scopes.length - 1; i >= 0; i--) {
                const scope = scopes[i]
                const foreground = tokenTheme._match(scope)._foreground

                if (foreground !== defaultForeground) {
                  return {
                    ...token,
                    scopes: scope,
                  }
                }
              }

              return {
                ...token,
                scopes: scopes[scopes.length - 1],
              }
            }),
          }
        },
      })
    })
  )
}

class TokenizerState implements monaco.languages.IState {
  constructor(private _ruleStack: StateStack) {}

  public get ruleStack(): StateStack {
    return this._ruleStack
  }

  public clone(): TokenizerState {
    return new TokenizerState(this._ruleStack)
  }

  public equals(other: monaco.languages.IState): boolean {
    return (
      other instanceof TokenizerState &&
      (other === this || other.ruleStack === this.ruleStack)
    )
  }
}

For some reason between those versions of the editor, the matched foreground (tokenTheme._match(scope)._foreground) has changed and is now returning a different foreground in some cases causing the syntax highlighting to be off.

I was going to file as a bug, but since there is no official guide and these are accessing private APIs it makes sense they are prone to breaking. Is it possible this is something on the roadmap or in the meantime there is a solution to workaround this until official guides/utilities might be available? I appreciate any help and understand there are a lot of moving parts here to make this all work.

Monaco Editor Playground Link

No response

Monaco Editor Playground Code

No response

[souporserious]

@alexdima sorry to ping, but do you have any advice here? I'm really stuck on this issue 😕 my latest attempt was forking the monaco-tm example and following your answer here, but I just get black text I think possibly because the colorMap is only returning [null, "#000000", "#FFFFFF"].

For context, I'm trying to create a Next.js plugin that sets up TextMate grammars automatically.

[tamayika]

I don't know this is suitable for you, but you can monaco-vscode-api to use textmate grammer for syntax highlighting.
This currently works on monaco-editor 0.34.x and work in progress for 0.36.x at here)

[souporserious]

That library looks suitable from what I can tell, thank you for the link! Looks like they just merged that PR 🎉 I'm going to play around with it and see if I can get something working.

[caleb1248]

I also struggled with the same problem, and I ended up making this thing. It works pretty well and I use it for most of my projects involving monaco-editor.