Lazy loaded shiki languages during syntax highlighting

[43081j]

Changes

There's a few changes in this PR, each contributing to memory usage improvement. Bundled into one PR since all but one are close to one-liners.

• Lazily load shiki languages if they're not already loaded
  • Rather than falling back to plain text, we attempt to load the language first
  • We default to ['plaintext'] rather than all languages, leading to all others being lazily loaded by default
  • This results in a chunky reduction of memory since we're no longer holding shiki's many language modules in memory for no reason
• Discarding contexts after build end
  • Some contexts were being held around permanently (i.e. retained in memory). By discarding them on build end, we allow them to be garbage collected
• Move static image helper into a factory
  • To avoid capturing local scopes, it has been moved into a factory
  • Since we store it on globalThis, we were holding deep references to large objects before
  • Now we no longer hold those refs, they can be garbage collected

Memory benchmark

Before:

	Elapsed time (s)	Memory used (MB)	Final memory (MB)
SSR build	3.11	91.74	128.05
Client build	0.11	-0.04	128.26
Static generate	0.91	4.88	132.53

After:

	Elapsed time (s)	Memory used (MB)	Final memory (MB)
SSR build	3.50	61.52	97.82
Client build	0.10	-41.66	56.20
Static generate	0.90	5.02	51.29

I did a few runs and got roughly these results each time.

Testing

• Extra test added for shiki highlighting via a lazily loaded language
• Memory benchmarking run to observe benefit of changes
• Full test run for regression

Docs

We may need to update docs on the fact that the default shiki languages are now ['plaintext'], although it shouldn't make much difference to a user since the rest will be lazily loaded anyway if they use them.

For reviewers

Each commit describes its own change, so it may be worth reading the commit messages and/or the diff of the individual commit if you want to understand better.

[changeset-bot]

🦋 Changeset detected

Latest commit: aaddd40

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[43081j]

making this a draft for now as its possible we can't have these nice things unless rehype lets us do async 👀

edit: false alarm, looks like it is happy

[bluwy]

Thanks! All these changes make sense to me. About shiki lazy load, we actually used to do the same but I removed it because I thought shikiji/shiki 1.0 already did that by default, but in hindsight it probably didn't because its highlighter functions are all sync anyways (which you pointed out).

About your message on Discord on version bumps, this will require a major for @astrojs/markdown-remark as it's published. Adding a major changeset for it should be enough and changesets will auto bump the rest (I hope). There's already many things within @astrojs/markdown-remark that had been marked deprecated, and this seems like the right time to clean it all up.

Would you be able to add a major changeset for@astrojs/markdown-remark? I'll handle the other major changes in another PR.

[43081j]

have fixed the broken test

FYI it was working because of this it seems:

astro/packages/astro/src/runtime/server/escape.ts

Lines 99 to 103 in 0be26d8

	// If a promise, await the result and mark that.
	else if (typeof str.then === 'function') {
	return Promise.resolve(str).then((value) => {
	return unescapeHTML(value);
	});

i suppose <Fragment set:html=${promise}> results in it awaiting the promise internally via the above, so it was working fine

edit: yeah of course, we await the child in renderChild so that was resulting in the highlight task being resolved properly

[bluwy]

Thanks!