How do we define a Tokenstream?

[RunDevelopment]

For #2070 I had to look at the tests of TokenStreamTransformer.simplify. And from these tests, a token stream is defined as:

type TokenStream = Array<string | Token | TokenStream>;

interface Token {
    type: string;
    content: string | Token | TokenStream;
    // some other properties
}

Yes, nested token streams.

But I always assumed that a token stream looks like this:

type TokenStream = Array<string | Token>;
// Token same as above

And I even added the condition that a token stream doesn't contain adjacent strings. I assumed this because the token stream produced by tokenize has these properties and because these properties are very useful.

Point is: I wrote code with these assumptions, so do I have to rewrite it?

What is the correct definition of a token stream?

/cc @Golmote @mAAdhaTTah

[RunDevelopment]

We might even want to narrow down the definition of Token to:

type TokenStream = Array<string | Token>;
interface Token {
    type: string;
    content: string | TokenStream;
    // some other properties
}

because that is what Prism.tokenize actually produces.

[mAAdhaTTah]

Yeah, in general, if we can bring the types into alignment with what is actually produced at runtime, I'm ok with tightening up the types. It would only mean that consuming code has been handling cases it doesn't have to.

[RunDevelopment]

Closed by #1782.