LVB Gridlines do not transit ConPTY (Win32 "underline", overline, left/right)

[mnefedov]

Windows 10, Windows Terminal, Version: 1.3.2651.0

Windows build number: [run `[Environment]::OSVersion` for powershell, or `ver` for cmd]
Windows Terminal version (if applicable):

Any other software?

I use

-- SetConsoleMode(hStdin, ENABLE_WINDOW_INPUT | ENABLE_LVB_GRID_WORLDWIDE )
-- I do "| COMMON_LVB_UNDERSCORE' on the attributes of CHAR_INFO 'line'
-- WriteConsoleOutput(... line );

I am 95% sure the rest of the code is valid -- because I can do colors, etc.

Expected behavior

-- characters should be underlined.

Actual behavior

-- no underline

[j4james]

The problem is that the console APIs don't have a way to set the real underline attribute. COMMON_LVB_UNDERSCORE is actually a separate concept - it's a grid line rendered at the very bottom of the character cell, and grid lines are not currently supported in Windows Terminal.

If you want your text to be properly underlined, you'd need to have the Virtual Terminal mode enabled, and use a VT escape sequence (\e[4m) to enable the underline attribute before writing out your text. You can then use another sequence (\e[24m) to turn the underline off.

If you really do need support for the COMMON_LVB_UNDERSCORE grid line in Windows Terminal, we'd have to come up with a way of forwarding that attribute over the conpty connection, and that's a bit complicated.

(Note to any core devs reading this: I've suggested before that we could possibly use the ECMA ideogram SGR attributes for this, but those attributes aren't very well defined, so I'm not convinced that's a great idea. It's something to consider though.)

[mnefedov]

Thank you James. I played with the escape seqs, and I got something working, but I have side effects after enabling Virtual Terminal Mode, I feel it is because my current rendering method is centered around WriteConsoleOutput(). I will debug.

[DHowett]

aren't very well-defined

Wow, yeah -- "underline or right side", "overline or left side"

I do think we should try to send these over ConPTY, and unfortunately I do think they need to be an SGR (specifically). They're a visual styling and terminals have a very good history of ignoring specifically graphical renditions they don't understand.

[j4james]

My understanding of the "underline or right side" and "overline or left side" is that those attributes were intended to be used with East Asian scripts when written in a vertical orientation. So an "underline" would be rendered to the right of the glyphs, and an "overline" would be rendered to the left. And that's the way those attributes are rendered by RLogin, so there is at least some precedence for interpreting them that way.

SGR 64 (the "ideogram stress marking") is less clear though. I desperately wanted that to mean a "proper name mark", which is essentially a form of underscore, but typically rendered lower than an underline, so that would make it a perfect alias for our grid line underscore. Unfortunately I think it's probably more likely to have meant an "emphasis point", which is usually rendered as a dot or circle.

That said, an underline is sometimes (rarely) used to indicate stress (at least according to this StackExchange answer), so it wouldn't be that terrible to interpret it that way. We unfortunately can't look to RLogin for precedence here, because they render SGR 64 the same as strike-through, and I don't think that's correct.

The bottom line is we could reasonably translate our right and left grid lines as SGR 60 and SGR 62. It would be more of a stretch to translate the underscore grid line as SGR 64, though, but not unthinkable.

[mnefedov]

James,

I had to change my code to NOT to use WriteConsoleOutput(), because it seems WriteConsoleOutput() "overrides" the underline. I am now using WriteConsole() in conjunction with SetConsoleCursorPosition(), and doing the ESC sequence when needed. This made the output very slow. I have not played with the cursor enable/disable yet.

[mnefedov]

... disabling the cursor for the duration of the output improves performance significantly, but it is still far behind the WriteConsoleOutput() performance.

[zadjii-msft]

I'm gonna re-purpose this thread as "make COMMON_LVB_*" attributes work in the Terminal - we definitely need to come up with something, because I'm sure that most legacy console apps are going to be using COMMON_LVB_UNDERSCORE for their underlines rather than VT.

[mnefedov]

FWIW using *_LVB_UNDERSCORE in CHAR_INFO.Attributes via WriteConsoleOutput() would be the nicest solution in my use case, because:
-- no need to track underscore attr,
-- no need to move the cursor -- performance consideration
-- no need to "touch" the cursor's visibility -- performance consideration
-- no need for a different render strategy for underlined chars -- code complexity consideration.

[j4james]

I came across something a while ago which I've been meaning to post, which could possibly work as a solution to this issue. I didn't think it was a great solution, but it's worth noting anyway.

Quoting from this site: https://www.cs.auckland.ac.nz/references/unix/digital/AQ0R4CTE/DOCU_006.HTM#I768

Digital UNIX enhancements for Asian languages include additional escape sequences for drawing and removing ruled lines in a specified area of a DECterm window. These additional escape sequences allow applications to construct tables and diagrams. The feature is a market requirement for Japanese

The main escape sequence they're referring to there is DECDRLBR (draws ruled lines on the boundaries of a rectangular area), which sounds like it was intended to meet the same needs as the grid lines in Windows.

If we were suggesting it as a VT sequence people could use in place of the legacy gridlines, it might be reasonable, but using it to forward existing gridline attributes over conpty seems like it would be horribly complicated. You could potentially be having to pass through a separate DECDRLBR sequence for every character you output.

Bottom line is it's probably useless, but I wanted to at least make a note of it.

Edit: The other thing I wanted to say was maybe we should just pass through the LVB_UNDERSCORE gridline as a VT underline, at least as a temporary solution, since that's assumedly the most commonly used. And I suspect nobody would really care that much that if it isn't aligned to the bottom of the cell. It's better than nothing.