support for SGR 21: double underlining text

[vapier]

the ECMA-48 (ISO 6429) standard (upon which all terminals are based) define SGR 21 to doubly underline text. Kitty supports an extension to SGR 4 where the subparam 2 does the same thing, but doesn't actually support the standard SGR 21. it should :).

[kovidgoyal]

SGR 21 has historically meant either bold off or underline (https://en.wikipedia.org/wiki/ANSI_escape_code#SGR_(Select_Graphic_Rendition)_parameters). Given that ambiguity I dont think it should be used at all. Do you know of any actual (reasonably widely used) client software that uses SGR 21?

[vapier]

the standard is clear here and has been since 1984 (~33 years). i don't know the source of that wikipedia line, but a glance at common emulators (like xterm and VTE) says they respect the standard.

especially when the alternative you're suggesting is the non-standard SGR 4:2 which few terminals support.

[kovidgoyal]

:) In the terminal emulator space, standards are pretty meaningless. See for example #227

And have you actually tried 21 in any terminal emulators? Try the following:

printf '\033[1mabc\033[21mxxx'

termite (vte based): Bold off
xterm: single underline
konsole: bold off
urxvt: bold off
kitty: no effect

So it's pretty clear where wikipedia gets that line from. In the real world SGR 21 means double underline, exactly nowhere, except in the standards document.

I really dont like this SGR code from a technical standpoint. It does not make much sense. The design of the SGR codes is codes from 1-9 set attributes and codes from 21-29 reset them. Using 21 as double underline breaks that pattern. So although the implementation is trivial (two line patch) I dont want to implement it unless there is a good reason.

And I am actually not proposing 4:2 as an alternative. That came from the VTE developers @egmontkob in particular. I just wanted to have single and curly underlines. For which sub-parameters were widely agreed as the best design.

[egmontkob]

VTE has just changed 21 to double underline last night. It used to have a comment /* Error in old versions of linux console. */ and then handled it as 22.

The idea for "4:x" for different styles of underlines came from you, I just proposed that then double underline should also receive its "x".

The pattern of 20+ turning off a feature is also broken elsewhere. Dim and bold are mutually exclusive (it's agian a legacy nightmare) and have the same code to turn off. Similarly, blink (5) and rapid blink (6) are both switched off by 25. Framed/encircled 51/52 are turned off by 54, overline 53 is turned off by 55. No point in insisting on the 20+ pattern too much :)

[kovidgoyal]

Yeah, but given the inconsistent behavior of terminal emulators about 21, no actual application can use it. Which means it is effectively dead in the water. So I dont see why it should be supported at all. IMO terminal emulators should simply ignore it. That way it is made clear to any poor application developer that might stumble upon it that it should not be used. Otherwise Alice will try it in her favorite terminal emulator and think it means bold off or single underline or double underline depending on the emulator and then later be annoyed when it does not work in other emulators.

[egmontkob]

@vapier I'm in the middle of filing an equivalent of #227 against chromium-hterm, but now that you've been notified about it anyway, I'm wondering if you could please do it for me? That would assign it to the correct component straight away which I cannot do.

[egmontkob]

We knew the choice of "4:3" for undercurl would break some terminal emulators, see neovim/neovim#7479 (comment), yet we decided that it's still the cleanest choice. There there was no existing standard to follow, we invented a new one.

For places where a standard does exist, like this 21 here, I believe that generally the best practice that leads out of the mess is to start following the standard (unless there's a solid reason not to, e.g. the standard is obviously bad, which is not the case with 21).

Note: ECMA-48 is a standard, Wikipedia is not.

[kovidgoyal]

The solid reason, is that 21 has different effects in different terminal emulators. So it is better at this point to ignore it than to implement it. Quoting myself:

Otherwise Alice will try it in her favorite terminal emulator and think it means bold off or single underline or double underline depending on the emulator and then later be annoyed when it does not work in other emulators.

[egmontkob]

Welcome to the beautiful world of terminal emulation :-)

[kovidgoyal]

urxvt also interprets it as bold off. So anyway, to summarize:

My vote is that SGR 21 should be ignored
@egmontkob votes it should be double underline
@vapier says it should be double underline
xterm says it should be single underline
urxvt and konsole say it should be bold off

Let's see what other people have to say. I am willing to follwo the majority decision.

[kovidgoyal]

Looping @leonerd into discussion.

[vapier]

i don't really understand your argument. "i don't want to follow the standard because other people don't do anything and i'm not interested in fixing things until everyone else does" ? "users can't rely on the standard SGR 21 because terminals don't implement it, so i'm not going to implement it because users won't use it" ?

doing the right thing is trivial to do now since you've already implemented everything else here. it's like literally 3 lines of code (or 2 going by the code style in this code base). it costs you nothing.

internally xterm def tracks the diff between underline & double underline. i don't know why it renders it as a single underline ... maybe there's some limitation in the UI stack.

the linux VT layer has historically made mistakes (including being buggy), and i don't think people normally hold it up as an example for other emulators to follow. plus with the rise of DRM/KMS, it's on the way out with replacements like libtsm/kmscon/wlterm replacing it in userland. i might take a stab at sending a patch to LKML anyways though and see how it goes.

[egmontkob]

xterm seems to interpret "double underline" as "one underline of double width".

[kovidgoyal]

Sorry that is not my argument. Kindly dont put words in my mouth. My argument is that, from the perspective of an application developer (the people that actually write programs to make use of terminal emulators), SGR 21 is unusuable today and will remain unuseable for the forseeable future. Therefore, in service to that poor application developer, it is best that terminal emulators ignore 21 to make it clear that it should not be used.

And blindly following standards, just because they exist, is not the "right thing".

[egmontkob]

@kovidgoyal Let's look at it this way: You wish to have double underline. Either you pick SGR 21 which has been defined for decades, and hope for the best (it'll work in xterm but not in kitty), or you pick the brand new 4:2 (it'll work in kitty but not in xterm). Why resist making it work at more places, even though that more places is still not everywhere?

[kovidgoyal]

@egmontkob Does anybody actually want to use double underline? The only reason it is included in the 4:x code is because you asked for it and I decided to implement it for the sake of greater harmony.

And 4:2, unlike 21 has the nice property that's its semantics are well defined and simple. If a terminal emulator supports it, it is double underline. If it does not, it is undefined. As for 21, its semantics are, if a terminal emulator supports it it could either cause a double underline or turn bold off. If a terminal emulator does not support it, it is undefined.

As somebody who actually writes terminal applications, it is obvious to me which semantics are preferable.

[egmontkob]

Does anybody actually want to use double underline?

I don't know. We'll see it with time. There are quite a few features that you cannot really tell in advance whether they'll succeed.

As somebody who actually writes terminal applications, it is obvious to me which semantics are preferable.

This is a rather philosophical question. Should it work at more places, even if it breaks somewhere? Or should we target fewer places only, but it should never break? You seem to vote for the latter, I don't think it's obviously the right choice. (And in the mean time again forget that emitting 4:2m does indeed break many emulators. Hopefully they'll fix it. Just as they could also fix not to misinterpret 21 against the standard.)

[kovidgoyal]

This is going nowehere, as I said, I am willing to follow the majority opinion. So lets vote. Currently two people have commented they want double underline and one has commented they want it ignored. I'll leave this issue open for a littel more time and anyone else with an opinion can chime in.

[leonerd]

For the record: I don't currently support other forms of underlining in libtickit - my terminal application library, so the issue is entirely moot there.

As to libvterm - I accept SGR 21 to mean double-underline, and I intend to keep that even if I add the second sub-parameter of SGR 4 as per the suggestions here.

I don't have a particular opinion on whether 21 ought to remain as a synonym for 4:2 or not. I think it's rare enough supported anyway that it's "best guess" as to whether it will work; I express no preference on whether you keep it or remove it.

[vapier]

thanks for that

[kovidgoyal]

You are welcome :)

[vapier]

my patch to update the kernel has been accepted by the tty maintainer. it'll prob show up in linux-4.17 if all goes well.

https://lkml.org/lkml/2018/1/29/858

[egmontkob]

@vapier Amazing, thanks! :)

[mintty]

I came across this issue while investigating about colon-separated SGR parameters for a catch-up of the mintty terminal with xterm. Just for information, mintty already supports these non-standard SGRs: 2 (dim), 3 (italic), 6 (rapidly blinking), 9 (strikeout), 11..19/20 (alternative fonts), 21 (double underline), 53 (overline) and 51/52 (framed/encircled) reinterpreted as emoji style preference.
The next version will support underline styles 4:0..4:5 (off/solid/double/wavy/dotted/dashed).
It will also support colour specifications in original ISO/IEC 8613-6 format 38/48:5:P 38/48:2::R:G:B plus xterm-double-colon tolerance 38/48:2:R:G:B, and including ISO/IEC 8613-6 CMY(K) formats 38/48:3:F:C:M:Y and 38/48:4:F:C:M:Y:K where F sets the 100% scaling value.

[egmontkob]

@mintty You seem to be the first to implement 4:4 dotted and 4:5 dashed – do I understand correctly from your wording that this is their exact assignment? If so, I'll make a note at the relevant bugs of a few other emulators, so that they won't implement them later in the opposite order :)

[mintty]

Hi Egmont, yes, I'd appreciate to align this assignment. Also the usage of the colour space parameter of colour modes 3 and 4, just in case anyone is interested to implement them too.

[vapier]

just to be clear, the SGR sequences you listed are all standard (ignoring the emoji bit). I'm not entirely sure what you are trying to say there.

when I added hterm support, I included 4:4 for single dotted and 4:5 for single dashed since those styles were already in CSS. I didn't mention it though as I wanted to see what the people did ;).

[egmontkob]

I didn't know hterm added 4:4 and 4:5 support, I'm glad hterm and mintty didn't end up assigning them in the opposite order :D

[mintty]

Good to have this aligned now.

the SGR sequences you listed are all standard (ignoring the emoji bit). I'm not entirely sure what you are trying to say there.

It had been considered somewhere else whether to reuse SGR 6 for curly underline as it would hardly be implemented. I wanted to make a statement that no defined sequence should be reattached and these are in fact in use.

[vapier]

all the SGR functions you listed, including 6, are clearly defined by the ECMA-48 standard (which i also referenced originally here). people shouldn't be suggesting or implementing conflicts with it.

see section 8.3.117 SGR - SELECT GRAPHIC RENDITION which defines SGR [0...65].

[christianparpart]

For the records, I just implemented 4:0 to 4:5 in my own terminal emulator, too.

With respect to the earlier discussion, I highly suggest to fix whatever software is broken (it's not just SGR 21) rather than erasing functionality, increasing confusion, or implementing it broken too (because others do).

All in all I am happy there is a place where devs can communicate and synchronize for the good and happiness of their users.