Consider support for Quarto qmd format?

[matthew-brett]

I wonder whether you would also consider supporting the new Quarto qmd format.

Why?

Because Quarto is looking like a very nice system for writing books and reports with excecutable documents, such as Jupyter and R notebooks. Qmd format is Quarto's cross-language text format for notebooks. It assumes .Rmd files are R notebooks, so Qmd format is the natural format for Jupyter notebooks in text form. We (@stefanv and I) are writing our new edition of a statistics book using Quarto. It would be a significant gain in usability if we could use the Jupytext / Notebook integration to edit and execute our documents in native Qmd format (although we are using Rmd at the moment).

What?

It's a slightly modified version of .Rmd format, where the cell metadata goes in comments at the top of the cell, as in:

```{python}
#| warning: false
#| echo: true
...
```

[mwouts]

Hi @matthew-brett , thanks for pointing out at Quarto. This sounds like a great initiative !

I've looked a bit at the documentation, and especially at the hello quarto example. That example reminds me a lot of the pandoc representation of Jupyter notebooks (the md:pandoc format in Jupytext). Do you know how the Quarto format differs from the pandoc format? (also I note that their example differs slightly from yours above)

Also I see that Quarto allows the user to edit qmd notebooks in JupyterLab, is this something that you tried out already? Does that address your point, or not? Please keep me posted!

[jjallaire]

Quarto is closer to the Rmd format than the md:pandoc format in that code cells use a modified variation of backtick code blocks rather than a pandoc div (:::). There are basically two axes (both of which have similarities to existing Jupytext formats):

1. Code cells use brace delimited language identifiers (e.g. ```{python} or ```{r}). This is similar to Rmd.

2. Code cell attributes are encoded as YAML within comments. This is similar to Myst (at least insofar as YAML is used, Myst doesn't use comments AFAIK):

   ```{python}
   #| warning: false
   #| echo: true
   
   1 + 1
   ```
   

We encode attributes within comments so that the metadata is easy to edit within existing Jupyter notebook UIs. So in many ways @mwouts you are correct that using comments means that we don't need to sync to .ipynb in all cases to be compatible w/ editing within Jupyter.

However, I think there are lots of cases where editing in both a plain text format (for more exotic markdown constructs like callouts, grid tables, layout panels, etc.) and within Jupyter will be valuable, which I think makes Jupytext integration very desirable.

It's on our short list to submit a PR for Jupytext .qmd support. If someone works on it in the meantime we are more than happy to chip in with code, suggestions, etc.

[mwouts]

Hello @jjallaire, thank you for joining the conversation!

Well I am glad to see the RStudio team bringing its IDE expertise to the Python world (if you didn't know, I started working on this project a few years ago when I had to switch from R to Python, and realized how hard it was to collaborate on ipynb notebooks compared to Rmd...)

So sure I'll be happy to help, and certainly we can add support for the .qmd format in Jupytext.

Regarding the actual implementation, we have a series of options:

1. Since quarto has a convert function, jupytext could simply quarto convert. That is how the pandoc:md format works. Advantages are: the converter is perfectly compatible with the official converter, and the Jupytext plugin is easy to code. The implementation is not as fast as a pure Python implementation, but in practice we've never had a complain about that (see the notebook Benchmarking Jupytext.py at https://mybinder.org/v2/gh/mwouts/jupytext/master?filepath=demo/ )
2. How is quarto convert implemented? If there is a Python API for it then we could use that API. That solution has the same advantages as the previous solution. This is how we provide support for MyST Markdown in Jupytext (and it is my favorite one)
3. Another approach would be to re-implement the Quarto format in Jupytext (like we did for Rmd). That is doable, but this will probably cause more maintainance on my side if the Quarto format evolves later.

Which one of the above do you think is best? Please let me know

[jjallaire]

Hi @mwouts, great to be here and cool to hear that your work w/ Rmd had some influence on Jupytext!

Of those options I think that quarto convert is the best way to ensure long-term fidelity/compatibility. quarto convert is currently implemented in Typescript so we'd probably want to just call it through the shell.

To really make this work well we'll want quarto convert to have some additional Jupytext awareness, and we may also want to add some functionality to RStudio to call Jupytext when saving .qmd files that are paired. As regards quarto convert, I'm specifically thinking of support for the tags that control code and output display (i.e. https://github.com/mwouts/jupytext/blob/master/jupytext/cell_metadata.py#L19-L34). We'll want to know that we are in Jupytext conversion mode and in those cases re-write the cell option YAML into tags (e.g. #| echo: false needs to become tags: ["remove_cell"]). I guess we'd also need to know whether to target runtools or jupyterbook.

Hopefully this all makes sense to you. We can add the requisite flags/behavior to quarto convert once there is a clearer picture about how this should all work end to end.

[mwouts]

Thanks @jjallaire !

Yes definitely I can prepare a PR that adds the .qmd format to Jupytext and uses quarto convert under the hoods.

I've been able to test quarto locally, how easy is it to install it onto the CI? Do I just need to run ./configure-linux.sh as for the local installation? Or do you recommend the debian package? (Ideally I would also like to add it to the Windows CI, if you had an example for that it would be great).

Also, should I expect a YAML header at the top of the file? When I ran quarto convert on the World population.ipynb notebook in the demo folder I did not get one. Usually in this header we put some information about Jupytext (version, format, version of quarto used), and also which kernel is used in the notebook.

Regarding the code and output visibility controls, I think we are still seeing some diversity in how this is handled in the Jupyter world. The notebook format now defines source_hidden and outputs_hidden (booleans), cf. https://nbformat.readthedocs.io/en/latest/format_description.html, but I don't know how well they are supported (and they probably deprecates the Jupytext mapping that you quoted above). Also I am aware that Jupyter Book uses tags (rather than booleans) like hide-input, hide-output, etc, cf. https://jupyterbook.org/interactive/hiding.html. Long story short, achieving identical look in different contexts might not be an easy task, so I'd rather keep that subject for later on.

Finally, you mention that RStudio could call Jupytext when saving paired .qmd files, that is an interesting point. By doing so we could guarantee that both the .ipynb and .qmd files are always up to date. Note that if you don't do that, the ipynb "looks" always up to date, because when the user opens it in Jupyter the content of the text file has priority over the actual inputs of the ipynb, and that might lead to people sharing the wrong version of the notebook (through version control or email) - until they save the notebook in Jupyter the .ipynb file remains out of sync. If we decide to do so, I'd like to implement a new flag in Jupytext CLI to convey the information that the .qmd file is the source of the update (currently jupytext --sync uses timestamps).

[jjallaire]

For CI I would just use the installers. The ./configure-linux.sh or ./configure-windows.cmd approach is fine too, but that's really for setting up a development configuration, running branches, etc.

It seems like there should at a minimum be a YAML header indicating the Jupyter kernel, that's likely an oversight (I also think that we should convert an H1 at the top to a YAML title). I will make those changes. For brevity Quarto supports a straightforward juptyer: python3 specification but we also support the more fully elaborated kernel definition + additional metadata so if you write all of that it should still end up round tripping (LMK if it doesn't).

I'm also thinking that it is likely that we could consider a --jupytext command line argument so we can have exactly the right behavior for this scenario. LMK how you'd like quarto convert to work in an ideal world and then I can either add that to the default behavior or condition on the --jupytext flag.

In terms of the saving scenarios, one thing I found less than ideal about JupyterLab was that when an unsaved notebook is changed on disk by Jupytext, JupyterLab doesn't automatically reload it (as many text editors do for text files). Do you know if there is any prospect of this improving (as it would make the multi-editor scenario much better).

[jjallaire]

@mwouts quarto convert now will always yield a metadata block when converting from ipynb:

quarto-dev/quarto-cli@1cf8c7f

This change is on main as well as in our v0.2.111 release available here: https://github.com/quarto-dev/quarto-cli/releases

If there is already jupytext metadata in the .ipynb then we will write something like this (from the World Population example):

---
jupyter:
  jupytext:
    cell_markers: 'region,endregion'
    formats: >-
      ipynb,.pct.py:percent,.lgt.py:light,.spx.py:sphinx,md,Rmd,.pandoc.md:pandoc
  kernelspec:
    display_name: Python 3 (ipykernel)
    language: python
    name: python3
---

However, if there is no jupytext metadata (not sure if this is ever the case when jupytext is being run?) then we write our standard abbreviation of jupyter: kernelspec: {} which is:

---
jupyter: python3
---

So you'd need to know that in some cases you might see this YAML rather than the fully elaborated version w/ kerenlspec.

[mwouts]

Hi @jjallaire , thanks for the fast iterations!

On my side I have prepared a new branch https://github.com/mwouts/jupytext/tree/quarto in which I have added the Quarto format. At the moment the round trip tests don't pass yet (tests.test_mirror.test_ipynb_to_quarto with quarto==0.2.111). I have seen two kind of issues:

1. After the round trip ipynb->qmd->ipynb the first cell becomes a raw cell
2. quarto cannot convert back notebooks for which I don't have a matching kernel locally (e.g. Error: Jupyter kernel 'julia-1.1' not found)
   Can you have a look at the branch and especially at that test?

In terms of the saving scenarios, one thing I found less than ideal about JupyterLab was that when an unsaved notebook is changed on disk by Jupytext, JupyterLab doesn't automatically reload it (as many text editors do for text files). Do you know if there is any prospect of this improving (as it would make the multi-editor scenario much better).

I completely agree! Well at the moment most of Jupytext is implemented in Python. If we wanted to reload only the text file we would need to 1. watch the file in JupyterLab (TypeScript) and 2. make the merge between the current notebook in memory (with outputs) with the text notebook again in JupyterLab (TypeScript) (the Python implementation is combine_inputs_with_outputs). At the moment I am not very skilled at TS so an external contribution would be very useful.

[jjallaire]

Excellent! I believe I have resolved both of these issues on main and in the v0.2.114 release (https://github.com/quarto-dev/quarto-cli/releases). LMK if things look correct on your end.

For JupyterLab, I'd honestly rather wait for core functionality that monitors edited notebooks for changes and automatically reloads them (or prompts for resolution if they are unsaved). This is definitely the right solution for users as it supports multiple-editors for notebooks in all scenarios (e.g. user has the notebook open in both JupyterLab and VS Code).

[mwouts]

Thank you @jjallaire! I have tested v0.2.116 and indeed it does solve the two previous issues.

Now we might want to improve speed (the round trip takes 3-4 secs per notebook, compared to 100ms for pandoc, and 10ms for the other formats).

Also, not all my sample notebooks are stable yet over a round trip. I think I am seeing consecutive markdown cells being merged (but I agree this is hard to avoid), and also raw cells are being converted to plain markdown cells. I'll leave that up to you (I can live with that and skip these sample notebooks in the tests).

There's one point for which I would need your input. Locally I have installed quarto with

wget https://github.com/quarto-dev/quarto-cli/releases/download/v0.2.116/quarto-0.2.116-amd64.deb
sudo apt install ./quarto-0.2.116-amd64.deb

Do you know how I can instead get the 'latest' version (as I'd like to install quarto on the CI)?

[jjallaire]

Great to hear! On performance, the issue there is Deno startup time -- modulo that we'd be sub 25ms. This has been an unpleasant surprise for us as our understanding was that Deno was intended for CLI tools (which clearly need to start up in < 50ms at the worst). We have one of two recourses here:

1. Figure out how to create a V8 snapshot for Quarto (which would more or less eliminate all overhead). Unfortunately this isn't currently supported so would be a bit of an adventure.

2. Have quarto be a stub to a persistent Deno process that we interact with using IPC of some sort.

We'll be pursuing these over the next few months so I expect the performance issue to be resolved in that timeframe.

Yes, Quarto uses Pandoc 'raw' markdown (e.g. ```{=html} to represent raw blocks so it's expected that these are lost during round trip. Similarly merging markdown cells is also expected.

I can think of two ways to get the "latest" version programmatically right now:

1. Hit this URL (https://github.com/gitapi/repos/quarto-dev/quarto-cli/releases) and then find the first release with prerelease: false. That will match what shows up as the "latest release" here: https://github.com/quarto-dev/quarto-cli/releases

2. git clone and call ./configure-linux.sh. This will give you the very latest dev version.

[cderv]

Do you know how I can instead get the 'latest' version (as I'd like to install quarto on the CI)?

Just chiming in this discussion to share other method:

• you can indeed use the API as mentioned above but get the latest release directly from https://github.com/gitapi/repos/quarto-dev/quarto-cli/releases/latest
  This method requires to parse the JSON response and get the url you want.
  Using curl and jq CLI tools are helpful (available in GHA runner I believe):

  # Get latest release json info
  latest=$(curl -s https://github.com/gitapi/repos/quarto-dev/quarto-cli/releases/latest)
  # Retrieve bundle download url (change extension depending of OS, here `.deb` for Linux)
  bundle=$(echo $latest | jq -r '.assets | map(.browser_download_url | match(".*[.]deb";"ix").string) | .[0]')
  # download bundle
  curl -L -O $bundle

  One liner

  curl -L -O $(curl -s https://github.com/gitapi/repos/quarto-dev/quarto-cli/releases/latest | jq -r '.assets | map(.browser_download_url | match(".*[.]deb";"ix").string) | .[0]')

• If available to you, using the gh CLI is the easiest (it is available in GHA runner)

  # download the latest release
  gh release download --repo quarto-dev/quarto-cli --pattern '*.deb'

Hope it helps

[mwouts]

Hi @cderv , thank you for joining the conversation, this is very helpful. I like the gh release download approach, however I am new to gh and I see I need to authenticate. Do you think this is really required for a download? If so, would you have an example that you could share? (the branch in which I've been trying to install quarto on the CI is https://github.com/mwouts/jupytext/tree/quarto )

If I take a step back, @jjallaire I think we're not far from being able to deliver a first version with support for .qmd files in Jupytext (both in the CLI and in Jupyter, allowing to open .qmd files as notebooks).

It is a good thing that you plan to address the latency of quarto convert, and thanks to our choice above you can work independently on that (although I believe that 2 seconds when opening/saving the file is not a blocker for early adopters).

Now there is one thing that is not included in the branch, it is the support for the .qmd extension in the Jupyter Notebook and Jupyter Lab extensions (allowing the user to pair the notebook with a .qmd file directly in the UI). Would you like me to do this now, or later when say quarto reaches v1 ?

[jjallaire]

We will 100% address that latency soon (it also comes up in a couple of places in the RStudio IDE).

I think it would be great to have the .qmd extension support in the Jupyter extensions now as I think that's a critical part of making the workflow seamless. Hopefully that's not too big of a deal to add.

[mwouts]

I think it would be great to have the .qmd extension support in the Jupyter extensions now as I think that's a critical part of making the workflow seamless. Hopefully that's not too big of a deal to add.

Yes sure - not a big deal indeed, I have added that in the branch.

I have also added a paragraph in the documentation, feel free to change it or expand it. If you don't mind I prefer to warn the users that the round trip is not always stable (as we try to make it stable for the other formats).

Finally, if you or @matthew-brett are willing to test the quarto development branch, you can simply use:

BUILD_JUPYTERLAB_EXTENSION=1 pip install git+https://github.com/mwouts/jupytext.git@quarto

(and then you'll need to restart Jupyter)

[jjallaire]

Hi @mwouts, yes it's fine to make a note about performance and stability for round-trips. We have started to take a deeper look at performance issues (denoland/deno#11916) and hopefully will have this much closer to 1 second very soon (and hopefully well below that in the not too distant future). Perhaps we should wait to merge this until we at least get the perf down below 1 second (so it will be less remark-worthy)

[mwouts]

Perhaps we should wait to merge this until we at least get the perf down below 1 second (so it will be less remark-worthy)

Yes good idea, you will let me know! We will just need to change the minimal version of quarto to be used inside Jupytext.

Also that will let the beta-testers to give us some feedback before we do the merge 😄

Also I still need to find out how to run gh release download on the CI (my latest attempt failed at https://github.com/mwouts/jupytext/actions/runs/1201826096)

[cderv]

Also I still need to find out how to run gh release download on the CI

@mwouts I think your workflow step should look like this:

     - name: Install Quarto
       env:
          GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}
       run: |
         # download the latest release
         gh release download --repo quarto-dev/quarto-cli --pattern '*.deb'
         # install it
         sudo apt install ./*.deb

I don't know if you can use gh with no auth activated, but with Github Action you can easily set it up because they provide a API TOKEN per workflow. You were close: you need to use the env: key to setup the secrets.GITHUB_TOKEN to the env var GITHUB_TOKEN, which is the only one known by gh CLI tool.

Let me know how it goes - it should work from my quick test.

[jjallaire]

An update: I've confirmed that the actual user code for quarto convert on World Population takes ~ 15ms so all the perf is overhead of one kind or another. I have managed to get this overhead down from 1.2s to 330ms by deferring or eliminating WASM loading so hopefully we are already in the category of acceptable round trip time where we don't need a special callout on quarto convert perf :-)

I am going to keep working w/ the Deno folks to see if we can compress that 330ms even further (not sure what mitigations/fixes are still available but we'll try). Will keep you posted on this and I think we will likely be okay to merge one way or another later this week.

[jjallaire]

Okay I don't think we can easily wring out much more that we have but hopefully the ~ 3.5x speed up we got will make things much more snappy (btw if you were calling quarto --version on every invocation as well it will be a 7x speed up if we can get rid of that, see my source comment on reading the version from the filesystem or perhaps even skipping this check to save the time).

I'll give a PR for tweaks to the Quarto language in the docs (will remove the perf qualifiers -- feel free to put them back in if you think they are still necessary). LMK your thoughts about the quarto --version thing.

[mwouts]

Thank you @jjallaire for the speed improvement and @cderv for the tip on the CI - I hope to find some time tonight to fix the CI as advised.

@jjallaire sure the documentation is yours, please go ahead, and thanks for addressing the issue!

Regarding the quarto --version I like to include a "format version number" in Jupytext files, this has been useful a few times to provide forward compatibility. But maybe either for quarto that is not necessary, we could fix the quarto version number to, say, 0.2, or alternatively we could call quarto --version less frequently, i.e. cache the result so that we don't call it every time a notebook is saved, especially from Jupyter. What do you think?

[jjallaire]

Yes, I think it's fine to fix the format version number at 0.2 (or even v1.0) and if there are ever changes we can increment it (note I expect very few if any changes to the core format).

There is another very inexpensive way to get the version (which is what we use in RStudio). This is described here: #846 (review). As noted, if there is no version file found it's the development version (which we by convention identify as version 99.9.9). This probably makes the case for serializing a static version number (as you don't want version numbers pinging back and forth based on whether you are using the dev version). In all cases though we definitely should avoid calling quarto --version for now as it effectively doubles the time required to save (I do expect to ultimately get the cost of this down close to zero but that won't likely be until next year).

[jjallaire]

Okay, scratch all of that about not calling quarto --version -- we are now short circuiting this in our wrapper script (reading the filesystem as described here) so calls to quarto --version are effectively free.

I still think that the format version number should probably be 1.0 as I think it's quite stable right now.

[jjallaire]

A couple of additional notes on version checking:

1. We have made this change in v0.2.136 on Mac & Linux but not yet on Windows (will come later today)

2. We have changed the output of quarto --version for dev version from "(Local Development)" to "99.9.9" (so it's always parseable as a version specifier)

[mwouts]

Excellent!

Here are the latest changes in the quarto branch:

• We now have a CI with quarto installed on Linux, thanks to @cderv (I'll now have to fix a few tests, and I'll also try to install quarto on the Windows CI)
• As suggested, I have set the quarto format version number at 1.0 (available when quarto >= 0.2.134 is found)
• I have taken your PR on docs/formats.md
• I also added a mention of the Quarto format in the README.md and docs/index.md, feel free to edit them, probably you would like to add a mention of the publishing system?

I can merge this PR and publish a new version (Probably Jupytext 1.12.0) when you give your go for it.

[jjallaire]

@mwouts green light from here (I think it's sufficient to have a more terse reference to quarto in the top-level docs so no need for edits there). Very happy that we've landed this, thanks so much for making it happen!

[cderv]

@mwouts Glad CI is working now!

I'll now have to fix a few tests, and I'll also try to install quarto on the Windows CI)

About this, for Windows, I maintain a Scoop bucket with Quarto manifest (https://github.com/cderv/r-bucket#quarto-cli). If you don't know Scoop, it is a package manager for Windows, which allow one to do scoop install quarto to have the last released version of Quarto installed. Scoop will deal with adding to PATH and manage version / updates. On GHA, this would mean: install scoop + add bucket + install quarto

However, you can also use the same as with Linux: gh release download + installing the msi file on Windows. That could be easier on GHA than installing scoop. I did not compare time.

This makes me wonder if we should provide an action to use in GHA to install Quarto. Probably worth it when first release will be out, in addition to dev version. 🤔

[jjallaire]

@cderv Even now I think it would be useful to have a GHA to install Quarto.

[cderv]

Ok, I'll work on something then. This will be a simple action with probably no parameter until it will make sense to choose a specific release, instead of development version.

[mwouts]

Excellent! So my plan is to ship a new version tonight, then.

With respect to the current branch I am just going to revert the tentative addition of quarto to the windows CI, and when the GHA for installing quarto becomes available I'll evolve the CI to use it on linux/mac/windows.

[mwouts]

The new version jupytext==1.12.0 is available on pip. Thanks to everyone, especially @jjallaire and @cderv for your help here!

(As a follow-up I'll try to install quarto on binder so that people can try it right away)

[cderv]

@mwouts

With respect to the current branch I am just going to revert the tentative addition of quarto to the windows CI, and when the GHA for installing quarto becomes available I'll evolve the CI to use it on linux/mac/windows.

you can now give a try to https://github.com/quarto-dev/quarto-actions and follow changes there.

[jjallaire]

@mwouts I am seeing what appears to be warning in the Jupyter console when I save (the save still seems to work):

quarto convert by default does not add ids to cells unless you pass the --width-ids flag. In the former case we write the nbformat_minor as 4 and in the latter (with ids) we write it as 5 so not sure where this warning is coming from (you will hopefully know better!).

Here's the commit where ids were added to the cell schema: jupyter/nbformat@9ede360

[mwouts]

Hi @jjallaire , I think this might be caused by an non up-to-date version of nbformat.

Can you run jupyter --version and report the version for nbformat?

I expect that the error should disappear with nbformat>=5.1.0, cf. https://nbformat.readthedocs.io/en/latest/changelog.html

[jjallaire]

Here is the output of jupyter --version:

jupyter core     : 4.7.1
jupyter-notebook : 6.4.3
qtconsole        : 5.1.1
ipython          : 7.27.0
ipykernel        : 6.4.0
jupyter client   : 7.0.2
jupyter lab      : 3.1.11
nbconvert        : 6.1.0
ipywidgets       : 7.6.4
nbformat         : 5.1.3
traitlets        : 5.1.0