Hotkey mapping on non-Latin keyboard layouts

[raphlinus]

This issue is a spinoff of #1040; the main scope of that was delivering key events. The implementation PR (#1049) does a reasonably good job with non-QWERTY Latin layouts, but does not handle non-Latin well, so that's what this issue is about. This analysis is largely based on Windows but similar issues exist across the platforms.

After a bit of research of existing application behavior, I found that when a non-Latin keyboard layout is selected, hotkeys (Ctrl-Z and friends on Windows) map ASCII keys (I think in the US layout, but see below). However, the current code does not do this - it queries VkScanKey to find which virtual-key corresponds to the codepoint for "Z", and that function returns -1 indicating error. Thus, the hotkey binding is dropped.

What is the correct behavior? Since ACCEL is based on virtual-key codes, I suspect what happens is equivalent to having a fallback mapping when VkScanKey fails, choosing 0x5A (b'Z') to represent the mapping of "Z". If this is the case, there are two problems:

• First, virtual-key codes are really only well defined for A-Z and 0-9. Others (period, comma, etc) are reasonably stable, while others (VK_OEM_[1-9], corresponding to virgule, octothorpe, brackets, and other ASCII symbols) tend to move around a lot. For example, I have no idea how you'd map a Ctrl-"#" shortcut, which would be Ctrl-Shift 0x33 in a US layout but Ctrl VK_OEM_2 in a DE QWERTZ layout.

• Second, if virtual-key codes are the source of truth for such mappings, then with our current types we've lost the ability for event dispatching to interpret the key the same as hotkey bindings, as we currently throw the vk on the floor (after perhaps using it to derive a Key value). The Web keyboard event has a "legacy" keyCode which in my testing on Windows seems to match the virtual-key code, which we might consider restoring. (I'll also note that keyCode is not stable across platforms, for example "#" in a German QWERTZ layout has value VK_OEM_2 on Windows but VK_OEM_5 on macOS, though the non-legacy codes are consistent: Key::Character("#") and Code::Backslash)

A second possibility is to use "code" rather than vk as the source of truth, and fall back to symbol positions in a US layout when the primary mapping fails. In this approach, when there is no key corresponding to "#", then the fallback is to find the key with Code::Digit3 and turn on the shift modifier. Basically, we'd have our own ASCII-to-code map based on a US layout, then on Windows use MapVirtualKeyEx with MAPVK_VSC_TO_VK to map the code back to a vk, and use the resulting vk in the ACCEL structure.

I am currently leaning towards the second approach, as it is based on more robustly standardized codes, and I see more clearly how to make it consistent across platforms. The main downside is that in edge cases it may not match the behavior of native Windows applications.

I'd love feedback from people who regularly use non-Latin layouts. Figuring out the right thing to do here probably mostly involves experimenting with a bunch of apps, and doesn't require any deep knowledge of Druid internals.

[raphlinus]

Let me propose for the sake of concrete discussion a heuristic for HotKey::match(key_event), which is really the second half of this discussion other than mapping to platform accelerator structures. Goals are:

• Reasonably simple.
  • In particular, don't require additional shell plumbing to query kb layouts or get vk's.
• Match the HotKey -> accelerator mapping in most cases.
• Fairly consistent across platforms, including web.

Non-goals are:

• Match platform behavior in all edge cases.
• Match the HotKey -> accelerator mapping in all cases.

Here's the heuristic:

• If the HotKey and key event are both Key::Character(), then apply the following logic:
  • If the strings are a case insensitive match, match. When comparing modifiers, accept shift in key if string is printable ASCII other than alphanumeric.
  • If the HotKey is a Latin character and the key is not, then match against the shift modifier + code of the key event, mapped by US layout. When comparing modifiers, ignore shift.
• If the HotKey is a Key enum other than Unidentified, compare key for equality, and the 4 basic modifiers.
• Otherwise, no match.

In some cases, the four basic modifiers are matched (shift, ctrl, alt, meta). In some, shift is accepted in the key event if it is not specified in the hotkey (specifying shift in the hotkey requires it to always be present in the key event). The other modifier flags (caps lock, etc) are not considered.

The predicate "is a Latin character" is defined as "a string consisting of a single codepoint in the range U+0020 to U+024F inclusive."

Here's a bit of rationale:

The reason for the case-insensitive match is so ctrl-[A-Z] works even when Caps Lock is enabled. Basically, for A-Z, we consider the shift modifier to be the source of truth, not case.

The reason for special handling of shift for non-alphanumeric printable ASCII is that the mapped character can be relied upon to capture the shift state. Thus, a hotkey spec of Ctrl-# will match Ctrl-Shift-3 in a US layout and Ctrl-# in a German QWERTZ. We should discourage (either through docs or warning) the specification of hotkeys with shift and ASCII symbols, as Ctrl-Shift-# will fail to match in a German layout. Note also that the correct hotkey for "+" is "+", not Shift-"=".

The rationale for U+024F specifically is that I believe both Western and Eastern European keyboard layouts are likely to contain mappings for ASCII symbols. Therefore, the other keys in this layout should not alias.

Here's an example of an edge case: In a Russian layout there will be more than one key combination that matches the hotkey spec Ctrl-",". For the former, both Code::Comma (ordinarily mapped to "Б") and Shift-Code::Slash will match. For the creation of the ACCEL structure, only one would get mapped (I'm assuming the latter but haven't tested it yet). Obviously the "Я" key will match a Ctrl-Z hotkey spec. I think this behavior is acceptable, if not absolutely ideal.

Another edge case: Ctrl-# will fail to match any key on the Web + macOS + Mozilla + US layout combination, as Mozilla reports Key::Character("3") for that combination (Shift + Code::Digit3). If we adopt this heuristic, we should fix our keyboard event processing so we report Key::Character("#") in that case, matching Safari (which I consider an authoritative source of truth) and Chrome.

If these are the two worst edge cases, I think we're doing well. Of course, it's possible I've missed something, so I'd love feedback.

[alerque]

I haven't studied this issue very closely in relation to Druid, but let me drop a side note here because I saw the idea of –if I understand it right– using raw key scan codes for bindings rather than the symbols they generate. In general this is a bad idea. One of my the most frustrating UI experiences I encounter stems from this. For whatever crazy reason browsers actually expose raw keycodes in addition to input characters. This has let to poorly thought out attempts by web designers to be "more accurate" that are just disastrous as an end user. Most commonly this shows up in form validations ... for example banks with entry fields that expect numbers that will only accept key scan codes that they think generate numbers.

In my case I use several keyboard layouts including a modified Dvorak layout (Programmers Dvorak) and similarly modified Turkish-F layout that has numbers & symbol levels reversed from QWERTY — by default pressing the keys gives me a symbol and using a SHIFT modifier gives me the Arabic numeral. The end result is that I cannot enter data in forms that reject input based on their expected scan codes.

This also turns up occasionally is desktop UI's in relation to hotkeys. When they try to be too smart and skip the OS's input methods and go straight to the source they are inevitably wrong for me and either the hotkeys are not what they say on the tin or it is actually impossible to enter them.

Please don't assume that just because 98% of people stick with QWERTY or other predefined layouts that it's okay to short circuit OS level input method tools. It's very rare that this is the right approach.

One exception where this can be the right solution is window managers. I have some WM bindings that are scan code base specifically so that they remain static across keyboard layout changes.