Migration to Sublime Text 4

[tajmone]

@pml-lang and @celtic-coder, I've just upgraded my license to Sublime Text 4. Originally I wasn't planning to jump version so quickly, but when the official new release came out I had a change of heart.

Anyhow, upgrading my license to ST4 has effectively invalidated my previous ST3 license, so I won't be running a copy of ST3 on my machine any longer — in theory it should be possible to have both versions running, but if I've understood correctly my old ST3 license won't work any more (I have to check this, I'm not sure if it just meant that I'm not eligible for updates or that license will be rejected).

The point is that I will be working on ST4 from now on, and won't be able to track whether I'm using features that are backward compatible with ST3 or newer ST4-only features. This might affect the user experience of anyone using Sublime PML with ST3 — but I suspect that you both have found that today your copy of ST3 was automatically updated to ST4 too, as in my case (at least, I had installed the DEV build, which always updates to the newer version, including MAJOR updates).

I'll be updating the package info to indicate the new ST4 builds from now on, since it makes sense to keep developing this package with future usage in mind, instead of investing on a product that has reached its end-of-life.

The distinction between ST3 and ST4 as two independent products is a bit misleading though. The new Sublime Text philosophy has dropped that distinction, and from now on there's only Sublime Text, and licenses are time-based and allow all updates during that period, including MAJOR version bumps.

So I guess that the Sublime PML package will follow the same faith, i.e. being constantly updated to the latest ST version, regardless of MAJOR version.

In theory, new major versions always support packages for older version, since new versions introduce new file formats to supersede the older ones, but are backward compatible with the latter — e.g. ST3 supports syntax definitions from ST2 (*.tmLanguage XML/Plist), so a package that contains a same syntax for both ST2 and ST3 (*.sublime-syntax YAML) will work with both editors, each one picking the latest supported version.

This means that in the future, when a new ST4 syntax format is released, Sublime PML could simply keep the current one for old ST3 users, and shift development into the newer format, and so on with each new MAJOR version.

Before making any changes that could break compatibility with ST3 I wanted to know whether you also found that your ST3 copy today automatically updated to ST4, and whether you're going to be using the new version or not.

As far as I know, you are the only two users of this package, beside me. So I want to make sure that any transitions to the new ST4 version will go down smooth with you. But, at the same time, I need to be able to make tactical decisions in preparation of submitting this package to official PackageControl channel, so that users will be able to install it from within the editor.

Since the package is already fairly usable, I think that after a couple of PML updates it might be ready for submission to PackageControl.

[pml-lang]

I wanted to know whether you also found that your ST3 copy today automatically updated to ST4

No, I'm still at ST3, version 3.2.2. I installed ST last month to explore Sublime PML.

whether you're going to be using the new version or not.

So far I didn't plan to update. I use IntelliJ IDEA for all my coding tasks, and sometimes VSCode. I will use ST (maybe ST4) for new PML documents I'll write in the future.

in theory it should be possible to have both versions running

Yes, according to Running 3 and 4 Side by Side, it should be possible.

[tajmone]

No, I'm still at ST3, version 3.2.2. I installed ST last month to explore Sublime PML.

So the automatic update to ST4 is only for those using ST3 dev builds. Maybe the logic at play here is that dev builds users are looking always for the latest version, including MAJOR ones.

I also noticed that dev builds for ST3 have been removed from the website now, which might also be the reason for the MAJ version update being enforced.

In my case, the update occurred at launch time, I noticed that it took longer to start up, and when the editor finally opened I immediately realized the UI had changed, but it I had to open the Help » About menu to realize it was ST4.

I will use ST (maybe ST4) for new PML documents I'll write in the future.

For the time being, ST4 can still be downloaded for evaluation without restrictions, as before. So it might be worth updating to it in any case since the website says that its fully backward compatible:

Sublime Text 4 is fully compatible with version 3. It will pickup your session and configuration automatically.

so you could benefit from packages updating to ST4. Many packages are already shifting to embrace the new features of ST4, so I think that very soon ST3 users will be stuck with old versions of those packages, since they won't be able to update them.

ST3 side-by-side ST4

Yes, according to Running 3 and 4 Side by Side, it should be possible.

Since the ST4 update occurred automatically on my machine, ST4 just took over the pre-existing ST3 data directory — and probably messed up some of its contents so that the folder might no longer work with ST3. I'm not entirely sure about this, but after the update I got a message that some packages were now broken and had to be uninstalled and re-installed to work with ST4, so I guess others were simply updated to ST4.

It's kind of annoying that I wasn't prompted for confirmation for a MAJOR version update (especially since it required a license update).

In any case, if I decide to keep also ST3 on my machine it would be a slimmer copy, with just a couple of packages that I might want to test for compatibility, so I might even use a portable version instead of a full installation.

[pml-lang]

it might be worth updating to it

I just started ST3 again and it asked me if I wanted to update:

I have now ST4 installed.

[tajmone]

On Shifting Sublime PML to ST4

I previously wrote that

In theory, new major versions always support packages for older version, since new versions introduce new file formats to supersede the older ones,

but I was wrong in this case because ST4 introduced new features in the .sublime-syntax format that can be enabled via the new version key:

When the syntax highlighting engine of Sublime Text requires changes that will break existing syntaxes, these modifications or bug fixes are gated behind the version key.

Currently there exist two versions: 1 and 2. The absense of the version key indicates version 1.

So, by setting version: 2 in Sublime PML I could enable and exploit the new syntax features, at the price of breaking backward compatibility for ST3 users. But I think it's well worth doing it, because the new features are quite powerful, and it would spare me from having the retouch the whole syntax in the future (it's already entangled enough, so it's no small job readapting it from version 1 to 2).

Syntaxes with Branching Contexts

The most notable new feature in syntax definitions is the ability to support branching contexts:

branch — Accepts the names of two or more contexts, which are attempted in order. If a fail action is encountered, the highlighting of the file will be restarted at the character where the branch occured, and the next context will be attempted.

branch_point — This is the unique identifier for the branch and is specified when a match uses the fail action.

The branch action allows for handling syntax constructs that are ambiguous, and also allows handling constructs that span multiple lines.

This would immediately solve a current problem with the [code node (#19) which is hard to handle with ST3 syntaxes. Branching context would make the whole PML syntax much smoother to develop and maintain because the PML language poses some challenges which can benefit from the new features.

Python Scripts/Plug-ins

Although ST4 by default supports the same Python version and API of ST3, it also introduces a new Python versions and API that can be used by packages. I think this was done to prevent Python scripts and ST3 package from suddenly stopping working, and that at some point in the future the new Python version and API will become the default.

As far as Sublime PML is concerned, Python scripts (plug-ins) would be used to handle complex key-shortcuts like Altb to wrap selections in [bold, etc. (something I was working on, but not yet in the package).

Again, it might not be worth creating them using the old API if the syntax is going to use ST4 features anyhow; so I might just as well take advantage of the newer API, which was greatly improved.

---

Sublime PML is still young, but since it's already usable I was planning to submit it to PackageControl in a couple of months. So the decision of whether to embrace the new features offered by ST4, or whether to stick to ST3 is a rather important one, both in terms of how it affects development/maintenance work, as well as the impact this choice might have on its potential user base.

Although I sporadically visit the Sublime Text forums, I'm unable to estimate how many users will be migrating to ST4 and how many will stick with ST3. Probably individuals are more likely to take the leap, compared to companies which would have to upgrade the software and license on all machines.

From what I can remember from the ST2 to ST3 update, the shift didn't take long, and soon ST2 simply became obsolete, and the packages that weren't updated to ST3 where either stale packages in state of abandonment, simple packages that wouldn't benefit from the new features (and still work on ST3).

[pml-lang]

I might just as well take advantage of the newer API, which was greatly improved

Agree 100%.