Incorrect display of characters written on top of the wide emojis

[o-sdn-o]

Environment

Windows build number: 10.0.18363.0
Windows Terminal version: 0.8.10091.0

Steps to reproduce

In PowerShell type two wide emojis, move the cursor back (eg: ESC[nD) to place it on top of or between emojis, and then type a narrow character over it
or
copy and paste the following:

"Place X at the end: 👨👨" + "X"
"Place X one   left: 👨👨" + [char]0x1b + "[1D" + "X"
"Place X two   left: 👨👨" + [char]0x1b + "[2D" + "X"
"Place X tree  left: 👨👨" + [char]0x1b + "[3D" + "X"
"Place X four  left: 👨👨" + [char]0x1b + "[4D" + "X"
"Place X five  left: 👨👨" + [char]0x1b + "[5D" + "X"

Expected behavior

"X" is placed as expected:

Actual behavior

The letter "X" is displayed incorrectly, moreover, emojis are unexpectedly shifted:

[o-sdn-o]

When this is fixed, it will become possible to correctly wrap the text line that contains wide chars
like this:

instead of:

[DHowett-MSFT]

Emoji cannot be split in half. Can you point to an example of a terminal that handles emoji in the way your "expected behavior" image reports?
Thanks!

[o-sdn-o]

Thank you for your attension!

I do not know any terminal emulator that would correctly display this case, but I believe that Windows Terminal should do this correctly, unlike others, and others will need to focus on it.

That is why I believe that it is "expected behavior":
When a user prints a regular character with preliminary indication of the coordinates for displaying in a terminal window, he expects to see it in the cell with these coordinates, and he cannot know in advance that a wide character is located at these coordinates and take this into account before displaying a regular character.

Also, taking this case into account when rendering the contents of its buffer is in the interests of the terminal emulator itself.
Since the width of the terminal window, as well as the cursor coordinates, is a multiple of the width of an narrow character, and not the width of a wide character, then only one cell can remain on the right border for displaying a wide character, and the terminal, in case of line wrapping, requires or a way to depict the presence of half of a wide character in this single cell, and display the second half on the next line.
The terminal displays a Unicode Replacement Character "�", when wide emoji, like 😋, wrapped to the next line, unlike CJK (which are also wide char) that are entirely placed to the next line.

There is no need to operate on the emoji halves separately, the halves are obtained in the output process only in certain cases, such as line wrapping or overlapping text of another high-level object, such as a frame with the message text over the already displayed text.

In the screen buffer, wide characters are usually represented by two adjacent cells, and if one of these two cells is overwritten with a narrow character (as it should happen when the text cursor is positioned manually and its new position falls just over the wide character), then the rendering subsystem should determine a separate way of rendering such a combination of cells.
For instance,
A wide character is represented in the buffer as:
👨 = [subcell1] [subcell2]
note, both subcells should have a reference to the whole grapheme cluster "👨" or be it with the exception of some attribute - the first part or second part, or they are entirely equal, but the first cell has a width=2 and the second cell has a width=0, to distinguish between them.
Narrow character:
X = [normcell]

Two cases of the cell combinations in the buffer:
[subcell1] [normcell]
[normcell] [subcell2]

first, the rendering procedure should draw a whole wide character in its two occupied cells, and then draw a narrow character on top of the wide one, given its position - left or right.

[o-sdn-o]

that's how it looks now

[o-sdn-o]

In addition, it seems to me that the features of using DECDWL / DECDHL and friends (#1884, Line Renditions, Double-Width, Double-Height Line https://vt100.net/docs/vt510-rm/DECDWL.html) lie in the same plane as visualization of wide characters, and causes the same rendering difficulties, but there may be other distortions of the visual representation during the flow of text.
These cases may be implemented with the same approach, as well as the visualization of wide characters on a cell-based coordinate grid.

[DHowett-MSFT]

I believe that the only correct way to handle a partial destruction of a double-width character is to remove the remaining half. There is no way for us to properly cut an emoji, or a CJK symbol that spans two cells, in half. Its meaning will be lost, and half of a character is not a unit that is representable in any encoding scheme or language.

RXVT-Unicode, which I believe set the standard for unicode use in terminals, treats your reproduction case as follows:

It doesn't support Emoji, but it does support a double-width glyph. Printing X over half of the double-width glyph destroys it.

The Windows Terminal currently has a bug where it does not destroy the double-width glyph. That, we should fix. I think it's tracked elsewhere, though.

[DHowett-MSFT]

The same applies to wrapping wide glyphs. They just cannot be broken in half.

[o-sdn-o]

Do I understand correctly that is possible to divide a wide character only by doubling the amount of memory allocated for one cell?

In that case, then the memory consumption can be neglected for the consistent visualization of all types of characters, since memory is not such a big problem these days, for example, tens of gigabytes of memory are needed only to compile the Windows Terminal.

[DHowett-MSFT]

I'm not concerned about storage or memory for wide characters. I'm concerned about how half-glyphs are incomprehensible and do not mean anything, and there is no prior art for cutting an emoji or an ideograph in half. 😄

[o-sdn-o]

It would be great to become pioneers 🚀 in this matter, since this problem exists.
No one cares about half glyphs, but text-based user interfaces cannot be rendered correctly without them in any way, and Windows Terminal is not least intended to display text user interfaces.

[o-sdn-o]

It is also possible that at some point when glyphs with a width of three cells (e.g.: 👨‍👩‍👦‍👦 Family: Man, Woman, Boy, Boy) are needed to support any specific scripts or complex grapheme clusters, the problem will become even more obvious.

[o-sdn-o]

I see no reason why the wide character should be completely destroyed when another symbol hits it.

I understand that it must be destroyed when copying the contents of cells with this state to the clipboard (until an Unicode standard will allow to form half glyphs by attaching the corresponding modifying codepoint), but such half cells in the terminal have full right to be displayed as is.

[DHowett-MSFT]

I'd like to understand this scenario a bit more.

Consider an application drawing a UI over top of a two-cell character.

• The two-cell character is split in half, and the half in the left cell remains on-screen.
• How does the application know that this terminal supports half-characters?
• When that UI is eventually dismissed, how does the application restore the missing half character? Remember: the terminal cannot save/restore the contents of the buffer for all applications reliably, so the application must keep track of it.
• If the application must keep track of it, and it must support terminals that do not have half-characters, it should move to the correct cell and re-print the entire character. That's the only way to ensure rendering stability.
• If an application must re-print the entire glyph, what is the value to the application? It must invalidate the same number of elements on the screen as it always has.
• If we standardize a new way for an application to print "the right half of a 2-cell character", it still must do feature detection on this terminal.
• How does this extend to characters that take up more than one cell? There is an almost infinite number of cell invalidation schemes that an application would need to be aware of.

Consider the case of an application wrapping a two-cell character at the edge of the screen.

At the right side of my screen, I see:

and on the left, I see

It does not seem trivial as a human to (mentally) reconstruct a symbol from an ideographic language split and moved to the other side of the screen. It's a readability disaster, and no other application anywhere (even complicated word processors!) implements wrapping that breaks a character in half. I'm sure there's a very good reason why.

I think we need to let UI libraries, ones that operate on pixel buffers instead of cellular text buffers, solve this problem for graphical UIs and not try to add single-cell occlusion to terminals to bring them closer to word processors.

[DHowett-MSFT]

(Thank you for taking your time to explain this!)

[o-sdn-o]

we need to let UI libraries, ones that operate on pixel buffers instead of cellular text buffers, solve this problem for graphical UIs and not try to add single-cell occlusion to terminals to bring them closer to word processors.

I do not agree with this, Unicode allows you to visualize UI elements as exquisitely as never before before the Unicode era. now a huge number of graphic elements are available in the form of separate codepoints, which would be a big mistake not to use this and not to do sophisticated programs in text-based mode.

[o-sdn-o]

There is an almost infinite number of cell invalidation schemes that an application would need to be aware of.

An application offering a user interface naturally has its own buffer, which contains a picture of the world, and this application must duplicate all transactions in the terminal into this buffer, and moreover, changes to the buffer must be recorded taking into account that the terminal may not support half-characters and other functionality, e.g.: even if the terminal does not correctly handle character widths, the application may always emit an ansi-sequence to correct cursor coordinates after the printing of each potentially wide characters, despite the awarnes of terminal capabilities.

[o-sdn-o]

Thank you for your clarification and for your work on the Windows Terminal, and I hope that my thoughts were at least a little useful 🙏🏻

[egmontkob]

I'm mostly with @DHowett-MSFT on this.

No terminal I'm aware of supports half of a double-width character (half-CJK or half-emoji). When one half is overwritten, I believe most terminals explicitly replace the other half with a space (whether retaining its previous background color, or taking the currently active one, I have no clue).

In addition to that, when the cursor is at the rightmost column (i.e. there's still room for a final single character in the row) and a double wide is printed, the new double-wide character is placed entirely in the next line, leaving an empty cell at the end of the previous. Ideally this empty cell doesn't get copy-pasted, and disappears at a rewrap-on-resize; and similarly, if the CJK would cross the line boundary after the rewrapped position then it's placed entirely in the next line. Just like with every decent piece of software. (On a side note, this is one of the reasons VTE forces a minimum size of 1 row, 2 columns.)

If there is a need for supporting half-CJKs and half-emojis (which I'm unsure about), the partial overwriting approach shown above is problematic for multiple reasons.

It assumes that the other (the one overwriting half of a CJK/emoji) is a single-width one, for which I don't see a guarantee. If there's a need to display half characters, the need for displaying two halves next to each other can arise very easily. There would also be a need to display a right half in the leftmost column, or a left half in the rightmost column, which would be impossible (and is a natural requirement for a text editor with non-folding lines and horizontal scrolling).

It doesn't allow flicker-free updates: in order to replace the half-emoji with another one, the other character might shortly be replaced by the undesired other half of the emoji.

It makes it necessary to come up with a logic that is very unlikely to be compatible with the current thinking and implementation of screen drawing libraries (e.g. ncurses, slang...). Not only would they also need to support storing half-emojis in their internal buffers, they'd also need to construct non-trivial ways of transmitting them to the terminal. E.g. if the right half of an emoji is visible, they'd need to reprint the entire emoji first, then move the cursor back, and then print whichever character hides its left half (which again might be the right half of an emoji, and then continue backtracking...).

This all falls apart uncontrollably.

If there's a need to display half-emojis, a new escape sequence should be invented which allows to place a half-emoji anywhere, without affecting any other cell. It could perhaps be a new mode to SAPV, or some brand new sequence. Once we have it (with agreement of a couple of leader terminals), it's perhaps okay to change the overwriting mechanism too as part of this story, to keep the other half present upon a partial overwrite.

There'd be a couple of other tricky questions. E.g. what to do on a copy-paste (do you just simply copy it? and how do you make sure not to copy it twice if the entire emoji is visible, maybe emitted by a screen drawing library as separate left + right halves?). Or what to do on rewrap-on-resize, how to decide whether an emoji (or a matching pair of two half-emojis) are allowed to be separated to different lines or not? Would the terminal need to store somehow whether the two halves are joined or not; what escape sequences and what other actions would modify this state, and last but not least, do we see a reasonable chance that various major terminals would come to a conclusion here? I'm afraid not.

It's an interesting and really hard technical challenge if and how this could be solved, in a way that's not specific to just a terminal or two, but is likely followed by the entire ecosystem (most terminals, screen drawing libraries, utilities). A proper solution is sure much-much harder than just coding in WT for a couple of hours to keep the other half there :)

[o-sdn-o]

use private codepoint (from Unicode Private Use Areas):
GCLPART: grapheme cluster left part
GCRPART: grapheme cluster right part
as a modifier character
(VT sequences are not suitable for copy-paste)

[egmontkob]

How would it work? Prefix for a single following character?

What kinds of problems would it use that it's not an escape sequence, but a regular character?

Or wait, you said "modifier character", like VS16 and friends, so would it come after the base character? Then what if there's a pause in the input stream after the base character? The terminal displays the entire glyph, potentially overwriting something, or overflowing to the next line, in turn potentially scrolling the entire contents, and then a bit later oops it should change its mind, undo these and place only half of it? Clearly cannot work, the special treatment has to be known in advance.

Also, private use means private use, up for you to use it in-house for whatever you want; not for Microsoft or any similar party to start assigning it a value (let alone a control instuction, rather than a glyph).

There's a whole lot more to the story, like how would it be compatible with wcwidth() and similar methods of determining a glyph's width in the terminal, etc.

Trust me please, it's not an area where you, or me, or anyone could such easily come up with a good solution.

[o-sdn-o]

How would it work?

like VS16. I suppose

what if there's a pause in the input stream after the base character?

If such a modifier appears first in the input stream the terminal should be triggered to text reflowing as in the case of window resize.

[o-sdn-o]

...not for Microsoft or any similar party to start assigning it a value (let alone a control instuction, rather than a glyph)

So they need to add these modifiers to the Unicode Standard.

[o-sdn-o]

to support double-height (DECDHL, 2x2 cells) characters use

GCNWPART: grapheme cluster north-west part
GCNEPART: grapheme cluster north-east part 
GCSWPART: grapheme cluster south-west part 
GCSEPART: grapheme cluster south-east part

which are preserve cwidth value of the base character.

[egmontkob]

Such adjacent user-perceived characters should be replaced by the application

By "the application" you pretty much every application in the world that supports Unicode (not restricted to terminal based ones), which would then have to be updated to your behavior. Absolutely hopeless.

The copy-paste activity is an area that is outside of the terminals.

Sorry if I wasn't clear, this sentence wasn't supposed to focus on copy-pasting, which is just a tiny part of the entire story. I meant this sentence for our main problem: half-cut CJKs and emojis. They are not a problem anywhere else, only in terminals. Trying to address this problem anywhere else (e.g. in Unicode) would create a giant headache for magnitudes more people than those who are affected by the problem in terminals. It's a wrong approach. Half-cut CJKs are a problem of the terminal world; if it is worth addressing at all (which I don't think is) then it should be addressed here, in the terminal world, not in some "upper entity". No one outside terminals wants or needs these.

[o-sdn-o]

...would create a giant headache for magnitudes more people than those who are affected by the problem in terminals.

There are many modifiers in Unicode today that can corrupt glyphs, but no one forces them to use and corrupt CJK. The application that will claim Unicode Standard compliance should ensure that the incorrect use of this technique does not go beyond the application.

...which would then have to be updated to your behavior.

Absolutely not, because at the moment there are successfully running applications that do not know anything about Unicode, or if they aware, but don’t know all aspects of its appliance.

Half-cut CJKs are a problem of the terminal world; if it is worth addressing at all.

This can be done with the optional behavior of whether or not to copy-paste halves when copying a selected area of ​​text or not. Let users decide what types of characters they want deal with outside the terminal.

[egmontkob]

So you imagine that thousands (millions?) of apps out there, which let's assume support a future Unicode version 15, will then suddenly no longer be fully compliant when version 16 comes out, or not fully compliant with whichever UAX or whatnot; and will have to be updated to at least prevent those homoglyph attacks, have reasonable copy-pasting, etc. All this for the sake of fixing a rare problem in terminals. Thousands of developers will thank you for that. (No.)

But let's also mention again that currently in Unicode all the modifiers are suffix characters, whereas in order to fix the problem in terminals we'd definitely need a prefix one, a suffix modifier that decreases the width literally cannot be reliably handled when the input slowly arrives over a stream. What do you think the chances are of Unicode accepting such a proposal? I believe it's pretty much zero.

On one hand I find the Unicode approach technicaly a bad choice, on the other hand, even if decided to go for it, I don't see a reasonable chance for getting buy-in from the Unicode folks.

Anyway, I believe we have to agree to disagree at this point. We've shared out thoughts with each other, and continuing this discussion doesn't seem to take us any further.

[Disclaimer: I'm not a WT developer and I'm not affiliated with Microsoft, not commenting on their behalf.]

[o-sdn-o]

Anyway, I believe we have to agree to disagree at this point. We've shared out thoughts with each other, and continuing this discussion doesn't seem to take us any further.

Thanks so much for the in-depth discussion. This is a lot of valuable information.

[o-sdn-o]

Another idea how to solve the problem of the allowing the screen drawing libraries (e.g. ncurses, slang ...) to display half characters to users.

Allow the terminal leaves the surviving half visible after splitting it, but only on the screen and only until the next repainting of the terminal window is occurred (text reflowing on some event of something else), and be sure to inform the application that all halves are “flushed”, and the application is responsible to reprint them (or whole screen) if they intended to do so.

Or, the application, at startup, tells to the terminal that the application itself is responsible for repainting entire terminal window when it is needed (by printing all screen lines, for example).

Some type of screen submode of the alternative screen buffer mode (CSI?1049h).

In this special mode, the terminal is not responsible to select text with the mouse, but must forward all mouse events to the application so that it takes care of the selection of text and placing data on the clipboard if the application provides for this.

Terminal should be responsible to emit VT-sequences to the application about the following data:

• window (pane) resize events
• dirty regions that need to be reprinted
• all mouse events, also events outside the pane if mouse is captured (any of mouse button is pressed inside pane before mouse cursor leaves the pane region)
• keyboard events
• window (pane) focus events

In order to be able to display the right half of the wide character along the left edge of the screen, it should be permissible to place the text cursor one character to the left of the left edge of the screen, location [y; 0]:
- CUP (HVP) y; 0
- CHA 0
- CUB 1 from [y, 1] position

To optimize application performance, the terminal is required to perform the following requests from the application:

• copy a certain region to the specified coordinates
• clear (fill) the specified region by specified color or by a character

[o-sdn-o]

Your arguments are very convincing, and I agree in the sense that the decision to use Unicode modifiers is like shooting a sparrow from a cannon, however, I can’t agree that it will do much harm to anyone.

One way or another, this solution is not feasible at a given time.

I consider the proposal to use a special terminal mode to provide a narrow circle of libraries using the terminal window for rendering the user interface more real and effortless, as well as the experience of using this terminal screen mode in the future will clearly show the need for standardization of half modifiers in Unicode, so I'm going to do make feature request this new terminal functionality.

[egmontkob]

Another idea how to solve the problem of the allowing the screen drawing libraries (e.g. ncurses, slang ...) to display half characters to users. [...]

First of all, your proposal allows to place half glyphs with limitations. E.g. there'd be no way to place a left half of X, followed by the right half of Y. What if an app distinguishes UI elements by different background colors, without an explicit character (e.g. line drawing) as borders, and two such items happen to be next to each other? If you need support for half glyphs, you need support for this, too. Also, screen drawing libraries need to know what to do if you create such a layout in their in-memory representation.

Second, I must point out that your proposal breaks plenty of things in terminal emulation:

• The emulation behavior is right now the same on the normal and the alternate screen, several apps even offer the users to pick one, expecting identical behavior. With your proposal, the behavior would be different, and those apps that don't switch to the alternate screen would have no chance of achieving your desired layout. You can't suddenly make the behavior different without potentially breaking things.

• You can't just say that moving the cursor to column 0 will no longer clamp it to column 1 but will actually move it offscreen, without worrying about breaking plenty of apps. You can't just say the counterpart of this for the right margin, either.

• You can't just say that all apps on full screen will have to from now on handle mouse themselves, and you can't forbid terminals from doing their own copy-pasting (e.g. for Shift+mouse).

• You can't suddenly change the existing model of how resize events are delivered, or how an application knows which parts to repaint.

and so on... Unix terminals carry a legacy of maybe 50-ish years, and at every step we have to be careful not to break things that were developed in this time. WT developers are in an much even harder situation, as they have to merge two different worlds without breaking anything. We have to be extremely careful with every step.

[...] so I'm going to do make feature request this new terminal functionality.

I'm sorry but I failed to understand this sentence, and I'm not sure where you're about to request this. In Unicode? If I fail to convince you not to push for your broken attempts then at least I sincerely hope that you'll point them to this discussion, so that they have a chance to read our (Dustin's and my) comments.

One of your screenshots shows Midnight Commander, an application I used to develop. It can use either the popular ncurses or the not-so-popular slang screen drawing library. I've been remotely following the development of these two libraries. There's not much happening, and they are really slow in catching up with changes of the terminal world. Even if terminals supported half glyphs, these libraries may not catch up for years (if not decades), and may even need backwards incompatible changes, I'm not sure about that. And then whichever app (e.g. mc) would also need to add support, which, given how its development goes, is again extremely unlikely. Before going ahead, it wouldn't hurt to hear a buy-in from these involved people, which at the very least requires the feature to be implemented by several popular terminals, which, in turn, definitely needs the feature to be well designed. The terminal world has way more popular and technically way easier features that are still not supported by many terminals, as well as these libraries.

It's an enormous amount of work to fix half-emojis, with necessary buy-in and coordination among many parties. It's extremely unlikely to happen. And even if it happens, it won't happen the way you think should, but the way experienced developers of the ecosystem will together design it.

To begin with, from the terminal's point of view, N years of experience with developing a terminal as well as bits of the surrounding infrastucture, I can assert you that the only reasonable way to start is to forget about changing Unicode, and go for an escape sequence that does one and exactly one thing and nothing else: prints a half-character somewhere.

Let me also tell you that N years of developer experience tells me that the terminal ecosystem has way more important problems to fix than this one. Also, fixing this problem would require multiple parties to care as much as you do, which I firmly doubt will happen.

I've been keep an eye on the bugtracker of many terminals and some terminal-based apps, as well as various public forum for years. I haven't come across any report about half-CJKs. It's not something Chinese/Japanese/Korean folks really care about, or really wanted to fix (or maybe they do prefer to see a space rather than a half-glyph? could be, I don't know).

[And finally allow me one comment that is not professional but personal:

Widespread use of emojis made this technical problem more prominient for a much larger user base. For me, the terminal is a tool for getting work done. It's not for playing games, it's not for watching videos, it's not really for having fun, and (some might disagree with me) it's not really for browsing the web, handling e-mail etc. either. There are other great tools for those purposes. In order to get things done (i.e. typical developer and sysadmin tasks), I couldn't care less about emojis.]

[o-sdn-o]

My point is that the moment has come when terminal emulators require a certain special mode of operation for the correct presentation of text user interfaces, this is primarily due to the fact that emojis, CJK and many other unicode ones came to the world of terminals. And these things are very difficult to push into the world of terminals, which was formed decades before.

GUIs are always haviest stuff, but TUIs are much lighter, and if the terminals would provide convinient functionality/framework for building TUIs on it, this will be a big leap forward in terms of user experience in cases where using the GUI is impossible or impractical for some reasons.

[egmontkob]

I wouldn't call displaying a half emoji, when there's no room for the entire one, instead of showing a space, a "big leap". I'd call it perhaps a minor cosmetic improvement, maybe even a questionable one (some people might prefer not to see glyphs cut in half).

Pair it with up with the fact that it's an enormous task to fix it. It needs a technical solution which has to be discussed, evaluated wrt. backwards compatibility, feasibility of implementation and whatnot. And then it needs to be implemented in many terminals (of which most likely hardly any cares), couple of libraries (presumably none of which cares), followed by each and every single application that cares.

I'm not saying I don't want this to happen. I'm saying I'm pretty sure it won't happen. You're the first person I hear caring about this story. You might find this important, others probably don't.

[o-sdn-o]

I wouldn't call displaying a half emoji, when there's no room for the entire one, instead of showing a space, a "big leap".

I don’t mean right now about half-characters, I mean, on the whole, the problem of displaying wide, complexly designed glyphs, working with the visible surface in the manner of graphic applications as a result of the possibilities of using true-color and various fonts at the same time. But all this graphic variety is strictly within the TUI.

[o-sdn-o]

Character segmentation and scaling

VT-sequence:
SCALE Nx; Dx; Ny; Dy
- turn scale mode ON and bind cursor coordinates with N and D
- D and N value range: [1-4]
- 1 byte/cell overhead in screen buffer

Unicode: 
- modifier name <GCSCALE1>..<GCSCALE256> (like VS1..256)
- in text: 
  <basechar><GCSCALE1..256>

Description:
- Storage: 
  byte = (Nx-1) + (Dx-1) * 4 + (Ny-1) * 16 + (Dy-1) * 64
- Meaning:
  e.g. 4x3 character in screen buffer:
  [1-4,3-3][2-4,3-3][3-4,3-3][4-4,3-3]
  [1-4,2-3][2-4,2-3][3-4,2-3][4-4,2-3]
  [1-4,1-3][2-4,1-3][3-4,1-3][4-4,1-3]
  e.g. 1x1 character: 
  [1-1,1-1]
  e.g. 2x1 character:
  [1-2,1-1][2-2,1-1]
  a) N <= D: select part N of character from D available and put it to single cell.
  b) N>D and D=any: stretch whole character on N cells
  c) N=D=0: reset scale mode to default (OFF)
- Coordinate binding:
  x0, y0 - cursor coord values when SCALE emitted
  x, y - current cursor position
  In case N<=D, Nx and Ny are function of x and y respectively: 
  Nx(x) = (x - x0 + (Nx0-1)) mod 4 ...
  Ny(y) = (y - y0 + (Ny0-1)) mod 4. ..
if (N < 0) N = 4 - N else N += 1 ...sleep 😕

This technique allows to work with characters ranging in size from 1 to 4 cells along both coordinate axes.

It doesn’t matter what size (cwidth) the character has, it allows put a wide character to a single cell if you want.

Output examples (VT sequence <SCALE;;;>)
- cout “a” produce  1x1 in buffer:
  [1/1,1/1]
- cout “😊” produce 2x1 in buffer:
  [1/2,1/1][2/2,1/1]
- cout “👨‍👩‍👧‍👦” produce 3x1 in buffer: 
  [1/3,1/1][2/3,1/1][3/3,1/1]
- cout “<SCALE1;1;1;1>😀” produce 1x1
- cout “<SCALE3;1;3;1>😀” produce 3x3
- cout “<SCALE1;1;1;1>👨‍👩‍👧‍👦” produce 1x1
- cout “<SCALE1;1;1;1>😀X” produce 1x1, 1x1
- cout “<SCALE2;1;1;1>😀X<SCALE0;0;0;0>H” produce 2x1(😀), 2x1(X), 1x1(H)
- cout “<SCALE1;2;1;1>😀🌎XH😀😀” produce 
  [1/2,1/1](left half 😀), [2/2,1/1](right half 🌎), [1/2,1/1](left half X) , [2/2,1/1](right half H), [1/2,1/1](left half 😀) , [2/2,1/1](right half 😀)

It is also possible with this technique to print out mathematical expressions and multi-level formulas (monospaced text documents with formulas, CJK, wide emoji and so on - are the Unicode problems that outside terminal world).

[o-sdn-o]

EDIT: 31 Jan 2020

Discussion is moved to the Terminals Working Group\Specifications\Issues.

I want to fill out the paper to a certain level, and then publish.

There will be no more updates here.

click to expand...

Character Segmentation and Scaling (Fractaling)

1. Abstract

This paper provides a general description of the solution to the problem of presenting and processing multi-sized (up to 4x4 cells) characters in a cell-based grid, where each cell one-to-one represents the visilbe part (fraction) of the character.

The following applications are related to multisize characters and character segmentation:

• Terminal Emulators.
• Linux console (when the X Window System is not running).
• Windows Console.
• Monospaced text documents.
• ... your suggestions

The solution allows to manipulate and store individual fractions of the whole character in a single cell for displaying them, as well as displaying multi-size characters in a cell-based grid and even allow their vertical splitting.
The solution also solves the problem of displaying wide characters in terminals by letting the terminal or the application running in it decide how wide the character will be, rather than relying on external data sources of these values that are subject to regular changes.

2. Solution

2.1 Definitions

Accordingly to the Unicode® Standard Annex #29, "UNICODE TEXT SEGMENTATION"

It is important to recognize that what the user thinks of as a “character”—a basic unit of a writing system for a language—may not be just a single Unicode code point. Instead, that basic unit may be made up of multiple Unicode code points. To avoid ambiguity with the computer use of the term character, this is called a user-perceived character. For example, “G” + grave-accent is a user-perceived character: users think of it as a single character, yet is actually represented by two Unicode code points. These user-perceived characters are approximated by what is called a grapheme cluster, which can be determined programmatically.

This paper defines a character as user-perceived character (or grapheme cluster).

2.2 Mathematical Presentation

To correctly display either a whole character of any size (up to 4x4 cells) or any selected character segment, only four numeric parameters Ps = Dx, Nx, Dy, Ny with range values of each from 1 to 4 are required.

Parameters

• Dx - count of parts along X-axis
• Nx - either width of the whole character or segment selector of the Dx available parts from left to right along the X-axis
• Dy - count of parts along Y-axis
• Ny - either width of the whole character or segment selector of the Dy available parts from top to bottom along the Y-axis

Interpretation

There are several cases possible (for each axis accordingly)

• D = 0
  • turn the scale mode off.
• N <= D
  • select part N of the character from D available parts and use it as a sinle-cell character (along the corresponding axis).
• N > D AND D = ANY
  • stretch the character to N cells.

2.3 Storing In Memory

Screen Buffer / Monospaced Text File

Each multisize character with a size of n x m that is greater than 1x1 is stored in the screen buffer (or monospaced text file)W x H as a matrix of n x m

Example:

3x2 stretched character "A" is located at x=3, y=2 in the screen buffer (of monospaced text file)

	1	2	3	...	...	...	W
1	...	...	...	...	...	...	...
2	...	...	A+Ps1	A+Ps2	A+Ps3	...	...
...	...	...	A+Ps4	A+Ps5	A+Ps6	...	...
...	...	...	...	...	...	...	...
...	...	...	...	...	...	...	...
H	...	...	...	...	...	...	...

A = "A"
Ps1 = { Dx=3, Nx=1, Dy=2, Ny=1 }
Ps2 = { Dx=3, Nx=2, Dy=2, Ny=1 }
Ps3 = { Dx=3, Nx=3, Dy=2, Ny=1 }
Ps4 = { Dx=3, Nx=1, Dy=2, Ny=2 }
Ps5 = { Dx=3, Nx=2, Dy=2, Ny=2 }
Ps6 = { Dx=3, Nx=3, Dy=2, Ny=2 }

Ps can be packed in one byte and overhead of screen buffer is 1 byte per cell:

byte = (Nx-1) + (Dx-1) * 4 + (Ny-1) * 16 + (Dy-1) * 64

Also there are only 256 variants for the Unicode modifier character value 0 - 255.

Characters with parameters N > D are not allowed to be stored in a cell-based grid. When such a character is to be printed to the grid, it must be segmented for each grid cell, and the parameters are recalculated for each filled cell.

2.4 Naming

2.4.1 VT-Sequence

Variants of the name for the VT-sequence

• Grapheme Cluster Scaling
• GCS
• GCSCALE
• GCSC
• ... your suggestions

2.4.2 Unicode Standard

Name of the Unicode modifier letter

• <GCSCALE1>..<GCSCALE256> (like VS1..VS256)
• ... your suggestions

3. Usage

3.1 Unicode Standard

Latest Unicode Standard defines three types of variation sequences:

• Standardized variation sequences.
• Emoji variation sequences.
• Ideographic variation sequences defined in the Ideographic Variation Database.

Only those three types of variation sequences are sanctioned for use by conformant implementations.

Accorginly to the Standardized variation sequences FAQ

Q: How can I propose a standardized variation sequence?

A: You can initiate the process of requesting a variation sequence by submitting an inquiry via the contact form. A thorough understanding of how Variation Selectors are used will make a proposal more likely to be accepted by the UTC. Read Section 23.4, Variation Selectors, UTR #25 and UAX #34, as well as the rest of this FAQ for background information. [AF]

Accodingly to the Section 23.4, Variation Selectors, UTR #25

A variant form is a different glyph for a character, encoded in Unicode through the mechanism of variation sequences: sequences in Unicode that consist of a base character followed by a variation selector character.

Variation Sequence

In a variation sequence the variation selector affects the appearance of the base character. Such changes in appearance may, in turn, have a visual impact on subsequent characters, particularly combining characters applied to that base character.

The standardization or support of a particular variation sequence does not limit the set of glyphs that can be used to represent the base character alone.

Placement in the Text

<basechar><GCSCALE1..256>

what if there's a pause in the input stream after the base character?

If such a modifier appears the first in the input stream the terminal should be triggered to text reflowing as in the case of window resize.

3.2 VT-Sequence

XTerm Control Sequences, Functions using CSI

Assing VT-sequence as a CSI/SGR command, because it define characters rendition state and sets the appearance of the following characters.

Human readable format
	ESC[ 110; <n1>;<n2>;<n3>;<n4> m

Sequence with "cooked" parameter
	ESC[ 111; <P> m

• n1, n2, n3, n4 are from 0 to 4.
• n1 = Dx, n2 = Nx, n3 = Dy, n4 = Ny.
• P = (Nx-1) + (Dx-1) * 4 + (Ny-1) * 16 + (Dy-1) * 64 from 0 to 255.
• SGR code 110: missing numbers are treated as 1.
• SGR code 111: missing number is treated as 0.
• 0 treated as reset the scaling mode (OFF).
• ESC[m (all attributes off) also resets the scaling mode.
• Instead of SGR codes 110, 111 suggest ... yours.

4. Expected Behavior

It doesn’t matter what size (cwidth) the character has, it allows put a wide character to a single cell if you want.

4.1 Unicode Standard

...

4.2 VT-Sequence

4.2.1 Printing

Output examples (VT sequence <SCALE;;;>)

- cout “a” produce  1x1 in buffer:
  [1/1,1/1]
- cout “😊” produce 2x1 in buffer:
  [1/2,1/1][2/2,1/1]
- cout “👨‍👩‍👧‍👦” produce 3x1 in buffer: 
  [1/3,1/1][2/3,1/1][3/3,1/1]
- cout “<SCALE1;1;1;1>😀” produce 1x1
- cout “<SCALE3;1;3;1>😀” produce 3x3
- cout “<SCALE1;1;1;1>👨‍👩‍👧‍👦” produce 1x1
- cout “<SCALE1;1;1;1>😀X” produce 1x1, 1x1
- cout “<SCALE2;1;1;1>😀X<SCALE0;0;0;0>H” produce 2x1(😀), 2x1(X), 1x1(H)
- cout “<SCALE1;2;1;1>😀🌎XH😀😀” produce 
  [1/2,1/1](left half 😀), [2/2,1/1](right half 🌎), [1/2,1/1](left half X) , [2/2,1/1](right half H), [1/2,1/1](left half 😀) , [2/2,1/1](right half 😀)

It is also possible with this technique to print out mathematical expressions and multi-level formulas (monospace textual documents with formulas, CJK, wide emoji and so on - are the Unicode problems that outside terminal world).

Line Wrap

...

Side effects

...

4.2.2 Capturing

...

5. Applications

5.1 Cost of Initial Implementation

...

6. Existed Infrastructure Compatibility

...

7. Security Issues

7.1 Unicode Security Considerations

Unicode Technical Report #36

This section describes some of the security considerations that programmers, system analysts, standards developers, and users should take into account.

For example, consider visual spoofing, where a similarity in visual appearance fools a user and causes him or her to take unsafe actions.

...

[jerch]

@o-sdn-o Can you open an issue in terminal-wg? This way more terminals devs will see it and it can be dicussed more in detail.

Some early remarks from my side:
As @egmontkob already pointed out - we have several issues with newer unicode rules in general in the terminal. While up to unicode 8 the simple wcwidth approach worked for most things (minus BiDi), the newer unicode versions introduced several new rules, that are not quite terminal environment friendly:

• grapheme clustering: currently not handled correctly by any common terminal or cmdline lib - this should be changed
• wcwidth is kinda broken / cannot deal with complicated clustering rules anymore
• different screen rendering possibilities given by unicode (like emojis can have a text/picogram repr, compounds can have different "run-widths") needs to be dealt with
• different unicode versions imply different rules, needs some kind of version "handshaking"

Among these issues the question whether a n-wide user percieved character can be split into n parts seems to be a minor detail. Until we have not fixed the basics it is hard to think about further extensions or alternate behavior.

Currently all terminals follow a common behavior here for n=2 (wider clusters were no issue until cluster rules took place) at the end of the row: if DECAWM (auto wrap) is set, the wide char would reflow to the next line early, otherwise it gets not printed (stays blank). Erasing an east asian wide char would actually clear two cells in most terminals.

Also note that your suggestion might not work with every terminal/render engine (not every terminal does the rendering on its own, technical level). Also to me it feels weird to be able to split those chars up (semantical level).

[DHowett-MSFT]

Thank you all for the very lively and very interesting discussion. I think @jerch is correct: the right place to continue talking about this is the Terminal working group.

[magiblot]

Excuse me for reviving this issue, but why all the fuss? The well-known Konsole terminal emulator has supported emoji left halfs for as long as I can remember, and I even took advantage of it in my Turbo Vision library:

2021-10-28.00-10-50.mp4

And unlike what has been said before, supporting this didn't require introducing a new escape sequence, extending the Unicode standard, buy-in and coordination among many parties, updating existing applications to prevent attacks or doing backwards incompatible changes (considering that the Turbo Vision API dates from the 90's). It was not an enormous task either, and it relies on whichever wcwidth is available on your system.

Of course, there are some edge cases that will never be supported this way: drawing a right half in the leftmost column, or a left half in the rightmost column, or a left half next to a right half, but none of these are necessary for what the OP initially suggested.

If Konsole supports left halfs, it is reasonable to request that Windows Terminal also does so. Not only that, but if left halfs are supported, support for right halfs is probably also feasible.

Note that while I think this, I also believe that Terminal's developers are free to decide they don't want to spend time on this and their decision shall be respected.

Cheers.

[o-sdn-o]

why all the fuss?

If a certain terminal emulator is limited only by the functionality of a text file viewer, then it is enough just to be able to display the left half.

If the terminal emulator wants to support raster operations on cell-based canvas, then it is forced to store/input/output glyph fragments, since currently glyphs have an arbitrary size in cells, and it depends more on the font used than on any standard ... Hence, it makes sense to discuss the format for representing a glyph fragment. The ability to store fragments will allow you to set the width of the glyph in cells, this can be done in exactly the same way as setting the foreground/background color, for example. It also solves the problem of setting the width of glyphs of arbitrary width, such as the "Family" emoji or a four-cell wide Devanagari syllable.

[jerch]

... why all the fuss? The well-known Konsole terminal emulator has supported emoji left halfs for as long as I can remember, and I even took advantage of it in my Turbo Vision library:

Then you rely on an implementation detail of konsole, that will fail on most other TEs. The "fuss" is/was needed to clarify, if it makes sense to get TEs into strict cell by cell drawing with half glyph rendering. While I understand the reasoning behind this, I dont follow the idea on a semantic and technical level. For the latter - some TEs dont control glyph rendering themselves, thus have no saying in this regard, the font renderer would just do as it pleases. The semantics are tricky as well - making such a mode the default one would change a well-established behavior for wide chars at the right border, thus is a no-go. Only chance I see here is an opt-in, maybe via a separate sequence as indicated by @egmontkob.

[o-sdn-o]

Emoji cannot be split in half. Can you point to an example of a terminal that handles emoji in the way your "expected behavior" image reports?

For the record and if anyone else is interested in this. Built-in vtm terminal (vtm --gui --run term pwsh) as an example of such a GUI terminal.

2x1 character samples:

"Place X at the end: 👨👨"           + "X"
"Place X one   left: 👨👨" + "`e[1D" + "X"
"Place X two   left: 👨👨" + "`e[2D" + "X"
"Place X tree  left: 👨👨" + "`e[3D" + "X"
"Place X four  left: 👨👨" + "`e[4D" + "X"
"Place X five  left: 👨👨" + "`e[5D" + "X"

3x1 character samples:

"Place X at the end: `u{2}हिन्दी`u{D0033}"           + "X"
"Place X one   left: `u{2}हिन्दी`u{D0033}" + "`e[1D" + "X"
"Place X two   left: `u{2}हिन्दी`u{D0033}" + "`e[2D" + "X"
"Place X tree  left: `u{2}हिन्दी`u{D0033}" + "`e[3D" + "X"
"Place X four  left: `u{2}हिन्दी`u{D0033}" + "`e[4D" + "X"

Special tailoring has been applied here (can be automated on the terminal side):

• U+00002 Grapheme cluster begin.
• U+D0033 Characted matrix selector as a grapheme cluster end (matrix width=3 and height=1 cells).

In general, table values ​​of character matrix selectors are used (up to 8x4 cell matrix):

• w: Character matrix width
• h: Character matrix height
• x: Horizontal fragment selector inside the matrix
• y: Vertical fragment selector inside the matrix

As a result, we get the following things in a text environment:

Complex scripts support, including RTL:

Various text transforms: