WindowEvent missing virtual_keycode while DeviceEvent contains it on Linux

[chrisduerr]

I just noticed on X11 that some WindowEvents do not contain virtual keycodes, while their DeviceEvents do contain them.

The events look like this:

Key(KeyboardInput { scancode: 5, state: Pressed, virtual_keycode: Some(Key4), modifiers: SHIFT }) }
KeyboardInput { device_id: DeviceId(X(DeviceId(3))), input: KeyboardInput { scancode: 5, state: Pressed, virtual_keycode: None, modifiers: SHIFT }, is_synthetic: false } }
Key(KeyboardInput { scancode: 5, state: Released, virtual_keycode: Some(Key4), modifiers: SHIFT }) }
KeyboardInput { device_id: DeviceId(X(DeviceId(3))), input: KeyboardInput { scancode: 5, state: Released, virtual_keycode: None, modifiers: SHIFT }, is_synthetic: false } }

What is interesting to me is that Shift+4 does not contain a virtual_keycode, while 4 does.

[murarth]

Well, the reason this happens is pretty straightforward: On X11, winit chooses the VirtualKeyCode value based on the X KeySym value of an event and there are two different functions to lookup keysym, which can both be found here:

winit/src/platform_impl/linux/x11/util/keys.rs

Lines 65 to 69 in 28f0eb5

	pub fn keycode_to_keysym(&self, keycode: ffi::KeyCode) -> ffi::KeySym {
	unsafe { (self.xlib.XKeycodeToKeysym)(self.display, keycode, 0) }
	}
	pub fn lookup_keysym(&self, xkev: &mut ffi::XKeyEvent) -> ffi::KeySym {

EDIT: Oops, didn't mean to submit this comment so soon.

So, for device events, keycode_to_keysym is used. While, for window events, lookup_keysym is used. If you look at the signature of each function, the reason for this is clear. lookup_keysym accepts a reference to ffi::KeyEvent, the X event that generates a WindowEvent::KeyboardInput. Device events can't use this function to get the same keysym because they don't have an ffi::KeyEvent to translate.

keycode_to_keysym returns a keysym corresponding to 4 for the device event because there is no knowledge of modifier state, only the keycode value. lookup_keysym returns a keysym that corresponds to $ (or whatever the user's keyboard indicates). There isn't currently a VirtualKeyCode variant for the dollar sign, so Shift+4 on window key event contains virtual_keycode: None.

Of course, the "why?" of this issue is not so clear, I think. What should the virtual_keycode value be for Shift+4? Should it be 4, Dollar (which doesn't exist at this time), or None? Should DeviceEvent have the same value or should device events be viewed as conceptually different from window keyboard events?

[chrisduerr]

Dollar is not a valid VirtualKeyCode, is it? Just like all other keys, the VirtualKeyCode should be the same regardless of the modifier pressed. It will just have an additional SHIFT modifier set. That shouldn't be much surprise though, since Key4 is exactly what the DeviceEvent delivers too.

Couldn't winit just use lookup_keysym and fall back to keycode_to_keysym if the initial lookup does not successfully map to a virtual_keycode?

It seems that Wayland at least manages to properly return a VirtualKeyCode here.

[murarth]

I'm not sure what VirtualKeyCode is intended to represent. For example, there are separate variants for Colon and Semicolon, despite these sharing a physical key on a typical US keyboard. In this case, winit reports Semicolon without Shift and Colon with Shift.

Should winit instead always report Semicolon? Or should we add VirtualKeyCode variants for all symbols (as #1266 aims to do)?

[chrisduerr]

It depends on the used layout afaik.

[murarth]

I'm not sure what you mean, but what I'm trying to say is, I agree that this issue is legitimate, but I'm not sure how to resolve it, as winit exhibits some inconsistency with regard to modifiers and VirtualKeyCode values.

If we're to solve this issue, I think we need to definitively answer this question: Should VirtualKeyCode values represent the "base" character of a key, ignoring modifier state?

• If yes, Shift+4 should have a keycode of Key4 in both DeviceEvent and WindowEvent instances. And, as per my above example, Shift+; should have keycode Semicolon instead of Colon (as it is for my keyboard layout).

• If no, we should accept adding keycodes for symbols such as !, @, #, $, etc. and Shift+4 should have a different keycode, e.g. $ (based on the user's keyboard layout).

I don't know the answer to the above question. It depends on the needs of applications. Do you have a preference toward either of the above scenarios?

[chrisduerr]

I'm not sure what you mean

I'm saying that special VirtualKeyCodes like colon usually appear on different layouts which do have these as base keys.

If we're to solve this issue, I think we need to definitively answer this question: Should VirtualKeyCode values represent the "base" character of a key, ignoring modifier state?

I think it would be nice to have both, however I don't think that's easily possible. So since this is not possible (we're not going to generate a VirtualKeyCode for Ö, right?), the only logical solution is to send it for the base key, not the "produced" value.

As an anecdote, Alacritty has had a long standing issue that key combinations like Ctrl++ to increase the default font size do not work because of different keyboard layouts. However, there's a simple solution for this: Allowing people to map keybindings to ReceivedCharacter events instead of raw keyboard input. This hasn't been implemented yet, but as far as I can tell, this would be the only way to reliably resolve this issue. Though it's important to note that this isn't super straight-forward either if you're including modifiers, since you not only need to track modifier state, but also take into account that while c generates the character c, ctrl+c generates \u{3}.

[nixpulvis]

I think it's a safe working assumption that there exists a keyboard layout out there somewhere with every permutation of shift/ctrl/etc modifiers with every possible character. Meaning while I have : and ; on the same key, someone else might have them on separate keys.

Can we inspect the keyboard layout for the keyboard device (do we do that already, sorry for my ignorance)? Otherwise, I feel like treating each key as a single name, with modifiers is the safest, and simplest option.

[chrisduerr]

This has been reproduced on Wayland too.

[kchibisov]

Dollar is not a valid VirtualKeyCode, is it? Just like all other keys, the VirtualKeyCode should be the same regardless of the modifier pressed. It will just have an additional SHIFT modifier set. That shouldn't be much surprise though, since Key4 is exactly what the DeviceEvent delivers too.

Couldn't winit just use lookup_keysym and fall back to keycode_to_keysym if the initial lookup does not successfully map to a virtual_keycode?

It seems that Wayland at least manages to properly return a VirtualKeyCode here.

Wayland works only because it uses a separate concept for this thing. it maps raw keys, and not keysyms, however I'm about to change that in #1534 , but it does that only for numbers, and not for the rest of the keys

If we're to solve this issue, I think we need to definitively answer this question: Should VirtualKeyCode values represent the "base" character of a key, ignoring modifier state?

If yes, Shift+4 should have a keycode of Key4 in both DeviceEvent and WindowEvent instances. And, as per my above example, Shift+; should have keycode Semicolon instead of Colon (as it is for my keyboard layout).

If no, we should accept adding keycodes for symbols such as !, @, #, $, etc. and Shift+4 should have a different keycode, e.g. $ (based on the user's keyboard layout).

I feel like we should add VIrtualKeyCodes for !,#,$, etc. We can't just assume querty layout and call it a day, since you can have different position for you Key6 and different different ^, so using keysyms for VirtualKeyCodes seems nice, since you can bind to thing like $ regardless of your layout, and your layout can have such thing without any required modifier.

@murarth
I'm fine with implementing Wayland/macOS(have it in vm now) parts of the proposed changes, however I'm not sure how much stuff should we add. For example all XKB keysyms are like ~3000 entries or something close to it. I feel like some basic US/EU keyboard symbols could be added and for the rest it doesn't really matter.

Also, I wonder what should we do about non latin layouts, since XKB has a special keysym for those things, so right now it's impossible to bind Russian A on X11/Wayland without scancode, so I wonder what should we do about such usecases. In theory we can have raw_keycode in addition to VirtualKeyCode, which will map raw codes directly to solve such problem. But I feel like it'll require more research to handle such things nicely.

I don't see any activity recently in winit, but I'm fine with maintaining Wayland backend, since I'm familiar with it the most and we need winit in alacritty, so I kind of have to.