Mismatched LaTeX support files detected

[wesleyburr]

By filing an issue to this repo, I promise that

• I have fully read the issue guide at https://yihui.org/issue/.
• I have provided the necessary information about my issue.
  • If I'm filing a bug report, I have included a minimal, self-contained, and reproducible example, and have also included xfun::session_info('tinytex'). I have upgraded all my packages to their latest versions (e.g., R, RStudio, and R packages), and also tried the development version: remotes::install_github('rstudio/tinytex').
  • If I have posted the same issue elsewhere, I have also mentioned it in this issue.
• I have learned the Github Markdown syntax, and formatted my issue correctly.

I understand that my issue may be closed if I don't fulfill my promises.

About 24 hours ago a previously working, unmodified Docker image that I use on a JupyterHub for hundreds of students started throwing xelatex update errors. It says the following:

... (usual stuff with a Quarto Render)
updating tlmgr

updating existing packages

compilation failed- error
LaTeX Error: Mismatched LaTeX support files detected.
(LaTeX)        Loading 'expl3.sty' aborted!
(LaTeX)        
(LaTeX)        The L3 programming layer in the LaTeX format
(LaTeX)        is dated 2023-08-11 but in your TeX tree the files require
(LaTeX)        at least 2023-08-29.

It then dies. For some cases, if the users wait for a few minutes, and then try again, it 'just works'. For others, it seems more persistent. Again, this is a Docker image that has been working fine for 5 weeks, with no issues. Similarly, I've had reports from ~ 10 students running Macbooks with TinyTex that this is happening to them as well.

This seems to be because when the Render happens, tlmgr and the packages are updated. But something recent has caused a mismatch in the supported/updated packages, especially with respect to expl3? Which may imply that this is just an issue with the library that the update is pulling from - but this is where my knowledge of the steps taken during Render breaks down.

Any thoughts? I'm not sure this is TinyTex in any significant way, aside from the fact that it is the TinyTex interface to tlmgr that is running the update, and taking a previously working installed distribution and somehow updating only part of it, so this mismatch happens.

[cderv]

It could be specific to how Quarto is calling TinyTeX and tlmgr when doing update. (as t seems you are using in Quarto context).

If you have easily access to your environment, you could try rendering a Rmd document to PDF with R which will use tinytex R package to call tlmgr. It is possible we run the right command and/or right order and maybe Quarto as an issue.

Which version of Quarto are you using ?

If you look for "Mismatched LaTeX support files detected." on the web, you'll see that this is common error that could happen when updating files. and usually solution is to run fmtutil-sys --all or fmtutil-user --all and tinytex R package does that for you I believe. (We have an internal function tinytex:::fmtutil())

Quarto does not seem to run this ever. @yihui from your experience, should this be run after an update ?
It seems we run this only when we install, but not when we update 🤔 so maybe that is not it...

@wesleyburr you could try running this command and see if it solves.

Some notes: Even if you are using Docker, if you are not using a full CTAN install or with locked CTAN repo, there will always be updates of packages or TeX Live itself, especially as Quarto is auto-installing missing packages. This type of error could happen for you because even if you update during one render, the TeX Live state come back to initial when you close you docker and re-run it. So maybe this cause issue with CTAN repo etc...

[yihui]

This issue was fixed in tinytex via 5946a2a, but Quarto has its own implementation of tinytex's features, and it might have missed this one. For tinytex, when it detects the mismatched expl3.sty (from l3kernel), it will run tlmgr_install(), in which we will run tlmgr_update() first because this l3kernel package has already been installed. In the end, tlmgr_update() will call fmtutil():

tinytex/R/tlmgr.R

Line 220 in cd41991

if (run_fmtutil) fmtutil(usermode, stdout = FALSE)

which fixes this issue.

[wesleyburr]

The image is using

quarto_1.2   tinytex_0.46

Very interesting to know that you did this update, and that it might not have propagated to Quarto.

Yihui, do you have any advice on how to freeze the tinytex install on an image like this, to prevent it from ever updating? This is used in a university setting, and it's frustrating to have a working VM like this for students, and have it decide to update randomly. I'd rather update the VM once every 4 months, and otherwise have it just be stable and unchanging.

[yihui]

For Quarto, you can disable the auto-install via latex-auto-install: https://quarto.org/docs/reference/formats/pdf.html#latexmk If you know in advance that your VM has included all LaTeX packages required for students to work on whatever they need to, I think it's definitely a good idea to disable the auto-install. In your case, it seems certain LaTeX packages were still missing, which triggered the auto-install (which failed with an obscure error).

I have hoped for a long time to write a tutorial about how to build, publish, and install one's own bundle of TinyTeX, but haven't found time yet.

[wesleyburr]

Amusingly, @yihui, I got access to one of the Docker images where this was happening. Make a .Rmd file, Knit to PDF, it successfully runs the tlmgr update, and then the Quarto renders fine. Which puts more evidence toward the hypothesis above that Quarto isn't correctly detecting this edge case and auto-correcting for it.

[cderv]

Quarto should now do the right thing (quarto-dev/quarto-cli@526d38a).

I got access to one of the Docker images where this was happening.

Are you able to confirm ?
This will require to install next pre-release build though (https://quarto.org/docs/download/prerelease.html): meaning Quarto > 1.4.395. No current pre-release has the change yet.

[wesleyburr]

I've regenerated our Docker image that all the students used, and pushed it live. I'll report back in a day or two. :)

[wesleyburr]

Things have been fairly stable with the workaround of "just run a R Markdown document, which forces tlmgr to work" for those students who haven't updated Quarto. However, yesterday, this stopped working. Attempting to knit a default Rmd file (create -> specify PDF -> save) results in this error in these tex-environments:

output file: tests.knit.md

! Undefined control sequence.
l.128 \cs_new:Npe
                    \__color_backend_devicen_colorants:n #1
                                  
Error: LaTeX failed to compile tests.tex. See https://yihui.org (etc.)

Any thoughts, @yihui? This one is entirely centered on tinytex and R Markdown, cutting Quarto out of the loop, but it's the same root issue - part of the tex distribution updates, and part doesn't, so the initial cause of the problem is the expl3.sty loading aborted. Saying

The L3 programming layer in the LaTeX format
is dated 2023-08-29, but in your TeX tree the files require 
at least 2023-10-10.

So my understanding of this is that the tlmgr update --all that runs automatically has updated some portion of the TeX tree, but isn't updating the L3 layer, and then the mismatch is causing the system to die. Is that correct? And do you have any idea WHY this is happening like this? It seems to come and go in waves for students. I just happen to have 800 guinea pigs with a variety of environments, so some do fine, and some crash into this issue hard.

[yihui]

Unfortunately, problems that are hard to reproduce are often also hard to fix.

I don't know how you install TinyTeX in your Docker image, but if you install the daily build (via tinytex::install_tinytex() or other methods), you shouldn't run into this type of problem, because we have already tried to compile several R Markdown documents and projects to PDF, and will publish the daily build only if it passes these tests. I can't think of a possible reason how TinyTeX could be partially outdated.

I think you could do the similar thing (although I still don't understand why it would be necessary): before publishing the Docker image, test it with PDF compilation, so you can be sure that you are publishing a working image.

If tlmgr update --all only updates some portion of the TeX tree and ends up with a broken TeX system, I wonder if tinytex:::refresh_all() could fix the problem of the outdated L3 layer.

[wesleyburr]

So the version that I push on Docker has those steps. I do compile a Quarto and Rmd doc to be sure as part of the Dockerfile, which runs tlmgr and ensures the TeX libraries are sync'd. It seems to be that if students have the image open for a few hours, and render a few times, at some point it decides to update ... something?

Of course the image is compiled at a set point (e.g., the last compilation was October 2nd), and then I normally just leave it alone. But the running of tlmgr on each render does seem to occasionally do something, and then the students end up in that situation.

I will test the refresh_all() and let you know. Thank you for the suggestion.

[wesleyburr]

refresh_all() doesn't seem to clear it for all folks, but a full removal of tinytex and reinstall does. Which presumably, as you mentioned above, pulls from the latest build, so everything gets updated.

quarto tools
quarto uninstall tinytex
quarto tools
quarto install tinytex

cleared up the issue for 6 different students over the last 6 hours. I wish I could tell you why things are desyncing like this. Basically, we've got about ~ 20,000 renders per week going on (800 students, each doing 15-30 renders to complete an assignment), and we're getting 1-2% of them that are running into this. So something is a bit unstable somewhere, and then it isn't able to fix itself without a purge and reinstall.

[yihui]

Looks like this was caused by a recent update of the l3backend package. Those students were not alone: https://stackoverflow.com/q/77291118/559676

I wonder if tlmgr install --reinstall l3kernel l3backend could fix the problem.

[yihui]

According to a TeX expert, tinytex:::refresh_all() is supposed to fix the problem.

[wesleyburr]

Got access to a broken machine today, running Mac OS 14, and can confirm that tinytex:::refresh_all() does indeed fix the problem, without needing to reinstall/update Quarto explicitly.

I guess the next question: is it possible to detect the mismatch error and automatically flag it for an R-level error message that suggests this? I'm thinking of your Issue 24 error that still pops up on fresh Mac installs, all these years later, to at least tell people how to get started in fixing it.

[yihui]

I have thought about your next question before you asked :) I just needed to confirm refresh_all() was the correct fix. Since you have confirmed it, I can certainly do it automatically. Thanks!

[yihui]

I spent quite some time on trying to reproduce the error "LaTeX Error: Mismatched LaTeX support files detected" in the original report, but failed with tinytex, which is because I have already fixed this problem in 5946a2a in 2021.

I think this issue was caused by Quarto, as explained at quarto-dev/quarto-cli#7252 (comment).

I'll commit a fix in tinytex soon to help TinyTeX recover from this "impossible" broken state automatically, but it won't help Quarto automatically because Quarto has its own implementation of tinytex::latexmk(), which currently needs a patch.