Add support for controller-based motion controls (gyroscope, accelerometer, …)

[pie75]

Describe the project you are working on

Non-VR Motion-Control-centric First Person Shooter.

Describe the problem or limitation you are having in your project

Godot does not support reading motion sensor output from non-mobile devices. (Android/iOS)

Describe the feature / enhancement and how it helps to overcome the problem or limitation

We would aim to extend godot's current Input format, either via core, or a module,
Adding Motion Controls: gyroscopes; accelerometers, as Vector3 in GDScript, and a separate class within the Input system, would be a natural way to handle this.
However, a more generic implementation would be to make available the individual motion sensors as axes within godot's existing axis system. These can later be combined into a Vector3, but will be available and function identically to an axis.
The developer having access to these hardware inputs via GDScript would make implementing motion controls natural.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

One way to do this is through binding an external library, such as JoyShockLibrary or the Input sections of Simple DirectMedia Layer, which has supported the same featureset as JSL since 2.0.14.
Alternatively, extending the godot core by replicating the features of those libraries natively is plausible.
Additional functions for managing these inputs, such as calibration, and 3D math, can be important, which can be managed by a module based around GamepadMotionHelper.

The Ideal scenario is one where a developer can simply handle the motion controls the same way as they would any axis, having both the actions system and direct access available to them, like this:
var p1_gyro = Vector3(get_joy_axis(Gyro_X), get_joy_axis(Gyro_Y), get_joy_axis(Gyro_Z)) or
var p1_gyro = Vector3(get_joy_gyroX, get_joy_gyroY, get_joy_gyroZ)
the Gyro_ structs or get_joy_gyro function might be just a binding for something like float JslGetGyroX/JslGetGyroY/JslGetGyroZ(int deviceId) from JSL, but it should function similarly.

If this enhancement will not be used often, can it be worked around with a few lines of script?

This enhancement cannot be achieved without changes to godot.

Is there a reason why this should be core and not an add-on in the asset library?

Matters dealing with hardware may be best implemented at core level.

I'm personally willing to work on developing this feature myself, but it's been a few years since I've programmed anything, and I'm not sure whether core changes or modules are the better way to go about this.
I imagine a combination of both has the most longevity. Separating GMH into a module is definitely more reasonable. However, one might argue that the entire gyro functionality might be superfluous for many users and should be modular.
I'm making this proposal mostly to gauge support and get some advice from godot developers on what to do first, before I start doing something completely wrong. Mainly whether module or core should be the focus.

Thanks for reading this far, and also thanks to JibbSmart for making so many tools for motion control development.

[JibbSmart]

Hi! I'm the author of JoyShockLibrary and GamepadMotionHelpers, created and continue to contribute to JoyShockMapper, wrote the best-practices for good gyro controls on GyroWiki, and extended SDL's gyro and accelerometer reading features to support Switch controllers.

I'd LOVE to see gyro and accelerometer as first-class input features in Godot. Gyro controls are increasingly popular, and provide a path to controls in games that are both drastically easier to learn and have far more room for mastery (compared with stick-only aiming controls).

I'm not familiar enough with Godot to propose an interface or how it should affect the current input system, but I'd love to help with implementation details. This is essentially what I did with SDL.

[NicholasShatokhin]

Does JoyShockLibrary support the PS5 gamepad?

[NicholasShatokhin]

I think, you need to add additional axises to Input options for gamepad like it done for the stick axises but for gyroscope and accelerometer.

Another option, the input event like mouse input events.

[NicholasShatokhin]

And please, allow me to use all three axises, not only x and y coordinate on screen. It will allow to implement amazing Overwatch-style controls (Z-axis is the same as X-axis but with much greater sensitivity)

Also, please add android platform because almost all modern phones have gyroscope and accelerometer too.

[Calinou]

@NicholasShatokhin Please use the Edit button (hidden behind the … dropdown next to your comments) instead of multi-posting.

[JibbSmart]

@NicholasShatokhin JoyShockLibrary does support PS5's DualSense, but I usually recommend SDL2, since it supports more controllers overall and will support the same Switch controllers as JoyShockLibrary soon (I've gotten a little involved there).

We'd definitely expose all 3 gyro axes and all 3 accelerometer axes. I like some of what Overwatch did, but there's lots of room for improvement there :)

[NicholasShatokhin]

I usually recommend SDL2

I tried to read gyroscope/accelerometer value using SDL2 in past year and I didn't find any way how to do it. Do you have a manual?

[Calinou]

Alternatively, extending the godot core by replicating the features of those libraries natively is plausible.

Note that this would be the preferred course of action, as Godot doesn't use SDL and linking against SDL just for gyro functionality is overkill. It would significantly increase the binary size, and keeping a low binary size is important to us.

(Copy-pasting code from SDL is fine, if you need to do that.)

[JibbSmart]

@Calinou Sorry, I'm a bit of an outsider on this one. How does Godot handle mapping across different kinds of controllers? For example, PlayStation, Xbox, Logitech controllers all report buttons in different orders, so the application can't actually tell if the "south face button" was pressed. SDL has an ever-growing database of controller mappings to solve this. If Godot doesn't already have a solution for this, we could kill two birds with one stone.

I understand wanting to avoid increasing the binary size (SDL's ~2MB is not negligible), but natively supporting Switch controllers, for example, is a lot of work.

@NicholasShatokhin 2.0.14 added:
SDL_GameControllerHasSensor()
SDL_GameControllerSetSensorEnabled()
SDL_GameControllerIsSensorEnabled()
SDL_GameControllerGetSensorData()

These currently work with PlayStation's DualShock 4 and DualSense, and the next update should have Switch Pro and Joy-Cons (mapped to the same coordinate space as PlayStation).

[NicholasShatokhin]

@JibbSmart Awesome! Thank you. I'm waiting for the next update :)

[Calinou]

Sorry, I'm a bit of an outsider on this one. How does Godot handle mapping across different kinds of controllers? For example, PlayStation, Xbox, Logitech controllers all report buttons in different orders, so the application can't actually tell if the "south face button" was pressed. SDL has an ever-growing database of controller mappings to solve this. If Godot doesn't already have a solution for this, we could kill two birds with one stone.

The SDL game controller database is already used by Godot: https://docs.godotengine.org/en/latest/tutorials/inputs/controllers_gamepads_joysticks.html

You can also pass custom mappings in the SDL2 format using the SDL_GAMECONTROLLERCONFIG environment variable or by calling Input.add_joy_mapping() in a script.

[JibbSmart]

@Calinou Okay, using just their database makes sense for basic controller support. SDL also exposes the touchpads on PS4 and PS5 controllers, and offers controller-agnostic access to controller lights (lightbar on PlayStation controllers, player number light on Switch controllers), too.

There's also JoyShockLibrary, which is smaller (by about a MB, maybe?) and only for reading from PlayStation and Switch controllers, but I started getting involved with SDL2 so I wouldn't have to maintain JSL anymore.

In principle, any developer should be able to easily provide proper support for standard console controllers on PC.

[Calinou]

@RetrocadeMedia Please don't bump issues without contributing significant new information. Use the 👍 reaction button on the first post instead.

As far as I know, nobody is currently working on gyro support. It's not easy to do either as it requires bespoke code for each controller model out there (and we can't link to SDL for size reasons). Therefore, I don't think it will be implemented anytime soon.

While I personally support initiatives to add gyro support to Godot (that are compatible with the above constraints), I don't have the knowledge to implement it myself. I've recently acquired a DualSense controller so I'm now able to test gyro functionality though.

JoyShockLibrary looks good to me at a glance, but remember that an implementation of this feature will probably need Linux support to be merged in Godot. It looks like the library currently doesn't build for Linux: JibbSmart/JoyShockLibrary#16

[JibbSmart]

Can we revisit the SDL2 size issue? Is SDL's 1.3 MB that big a deal?

Gyro controls are increasing in popularity. Valve is pushing gyro as the best mouse-like input on its Steam Deck, and Epic has embraced gyro in Fortnite on every applicable platform (everything except Xbox), making a big deal out of my involvement. As more and more players get into gyro controls, they're finding that's the best way to play with a controller.

More AAA games have their own gyro implementations in the works, but I can't get into specifics. Suffice it to say the trajectory of gyro support points to it being seen as a standard way to play before long. They're already supported on everything except Xbox, and the fact that competitive games like Fortnite are offering more powerful options on every other platform means Microsoft is probably exploring introducing motion sensors to their own controllers.

It seems to me that good, cross-platform, standardised gyro and accelerometer support (as provided by SDL2) is an important part of future-proofing Godot's controller support.

[Calinou]

Can we revisit the SDL2 size issue? Is SDL's 1.3 MB that big a deal?

If there is no way around linking the entirety of SDL2, then gyro support will have to be an extension, not a core feature. But first, I'd look at whether copying the relevant code from SDL 2 is a reasonable avenue. A similar process was followed for battery status support a while ago (which was since removed as it was incomplete).

On the bright side, since Godot already has its own gyro support code for Android and iOS, we could probably link against SDL2 only on desktop platforms. I suppose this would also make buildsystem maintenance easier (which is another cost of adding a large library to Godot core). We need to make sure a similar feature set can be exposed this way though, so that gyro controls created for desktop can be easily translated to mobile and vice versa. But even then…

Remember that Godot's binary size has been steadily increasing over the years. Godot 4.0 is already slated to be much larger than Godot 3.5, both for the editor and export templates. Keeping this default size low is critical on mobile and web platforms. Even on desktop platforms, people don't like to ship huge binaries for desktop applications. Many people like Godot for its ability to create desktop applications with a lower footprint than Electron, and I'd like it to remain this way 🙂

We also want Godot to remain fast to build from source (and master is already much slower to build compared to 3.x, unfortunately). This isn't just about initial builds either. Since builds are statically linked, every library will add to the link time, making incremental builds slower.

While it is possible to build Godot with specific features disabled, this is not a route we should expect most users to follow as it has a lot of potential complications.

Lastly, linking to third-party libraries means we have to cater to decisions that are taken upstream and may not play in our favor. This happened with Bullet physics shifting its focus from games to robotics and simulation.

[pie75]

I agree that if SDL2 must be linked, it would be best as an extension.
While Android and iOS include motion libraries, these do not apply for controllers connected to those systems. They only read onboard sensors (which is also possible on windows devices.) In practice, the system-provided libraries for controllers in Android and iOS are xInput level featuresets, and thus very limited.
Inevitably, for controller IMU support, SDL2 would have to be included in those builds.

However, I think it's far more ideal to have this support be a core feature, if possible.
Making using the sensors as straightforward as them having their own category, much like axes and buttons, would be the most user-friendly result. Having it be easily translated to onboard sensors would also be ideal. Jumping through hoops to make onboards do the same thing as externals is annoying, to say the least.

I ran into several issues when trying to reimplement SDL2's code in godot, but the most significant among them (besides my own lack of skill) was that, at least on 3.x, in Windows, it seemed to require what amounted to a complete rewrite of how controllers are structured in godot.

Currently, the core input system in godot seems limited to the scope of its reliance on Direct Input & XInput. Migrating everything on Windows to RawInput might simplify core adjustments like this - but to me, it seems like a very delicate, complicated process.
It's not something I felt confident doing (though I did try), so I experimented with extensions, modules, using either JSL or SDL2. But I couldn't get anything that felt right, in terms of implementation.

Ultimately, I kept scrapping implementations that I wasn't happy with, and frustrated with my inabilities, I decided to put any work on hold until I could work in 4.x
Now that I've got a vulkan-capable computer, I think I can feel a bit more motivated to work on this.
But ultimately, I'd be stuck working purely on the Windows side, which isn't ideal, even for an extension.
Rewriting the core input of this engine is probably not a task someone like me should be doing, certainly not alone.

On an ending note...
It's unfortunate, but no matter what approach is taken, it'll be some level of hackjob, bloated, and painful. We're in a time where the driver API's for these OS have long been insufficient, and consequently, the drivers for these devices are insufficient. So we're forced to extend the applications far lower level than they need to be, in order to account for missing driver functionality.

There's going to be sacrifices, and it's probably important to weigh which sacrifices are least impactful.
Several years down the line, all the work will probably need to be discarded, as the driver situation hopefully improves. I personally think that it's probably important to plan for that, as well.

[pie75]

With the relatively near future, June 2022 public release of GameInput, the latest refresh of Input API's on PC, available in Microsoft's Game Development Kit, It seems that the driver situation may improve more quickly than I imagined.

I imagine that eventually; given GameInput's status as a superset of the previous DirectInput, XInput, and Windows.Gaming.Input; and that GameInput will be a standard and redistributable library on all Windows installations back to Windows 7: Godot might be incentivized to migrate its Input backend on Windows to GameInput. Assuming that to be the case, the inclusion of a MotionInfo and MotionState Structure significantly simplifies the backend implementation of Motion Controls. (at least in theory)

This doesn't directly change anything on the other platforms. However, given Apple's recent push into Gaming (which I personally think is going to be as brief as it was decades ago, but let's assume not), and their apparent patenting of game controller concepts, we might see a similar-ish API implemented on Apple devices, like how everyone managed to copy Xinput.

Regardless, the simplification of the Windows Backend would make reimplementing the SDL2 code much simpler, and more interchangeable with any raw HID-dependent implementations on other OS platforms.

All of that is in the future, though.
At the present, I do not think that migrating to GameInput is reasonable, nor do I think it would necessarily help accomplish this goal. We may have to wait for community-driven solutions to make older controllers compatible with GameInput, time will tell.

But it seems to me, that future isn't too long off, and that means that any current focus on this issue should be on an Extension, despite the possible interface gripes that might bring; because an Extension would be easier to deprecate, when/if the GameInput API is implemented.
Native implementation of any kind, at the moment, would be more difficult to remove, even re-implementing SDL2 or integrating a JSL/JSL-like single header library.

That's my current stance, at least. Notably, it does make a strong assumption about the future of GameInput and Godot. I understand it's not a given that Godot should migrate to GameInput.
I hope to see more opinions from people more familiar with the other Operating Systems, who can give a better idea of what resources for solving this issue are available to them.

[Calinou]

That reminds me, it should be possible to use Steam Input to provide gyro controls in Godot games as long as they're running through Steam. This can also be done for games that aren't distributed on Steam by adding those games as non-Steam games in your library.

This solution isn't ideal as it requires you to start Steam to play your game (even if the game in question is DRM-free), but it's good to keep in mind.

[NicholasShatokhin]

@Calinou Maybe, the better solution is to detect is Steam Input available in use it if yes. In other case use other method.

[Calinou]

We discussed this proposal in a meeting. We concluded that for now, it may be better to improve GDExtension to allow for plugging additional input systems. This way, a SDL2-based gyro system can be developed independently of the core engine.

Having gyro support as a core feature will likely have to wait for input APIs to be better standardized. After using gyro aiming for a bit, I can attest it's definitely useful, and I do want to see built-in support of some kind in the long term. In the meantime, we can still document using Steam Input and/or JoyShockMapper with Godot in the documentation.

Edit (August 2022): This tasklist may be a good starting point for working on a custom implementation of accessing DualShock/DualSense motion sensors on Linux: RPCS3/rpcs3#12503

[JibbSmart]

Improving GDExtension for additional input systems sounds great :)

Regarding standards, I think it's probably pretty straightforward. Gyro input is angular velocity and it should probably be in radians per second. Accelerometer input is linear acceleration, and should probably be either g units or metres per second per second. Conversions between each are simple, of course. Axes should probably match whatever's the norm in-engine. JSL and SDL2 both use the PlayStation coordinate space (Switch controllers are converted to match). Conversion between different coordinate spaces is simple, too.

A sensible interface will work pretty much the same as far as the user is concerned regardless of whether it's backed by SDL2, JSL, Steamworks, or the official PlayStation or Switch libraries.

[VSofficial]

How about adding support for Adaptive Triggers in PS5 Controller/DualSense, since it's also one of the modern input and feedack offered. But one big thing is, most of the developers think and feel it as extra and needless feature for most of the gameplay offered, infact sometimes users switches off vibratons, adaptive triggers

[JibbSmart]

@VSofficial I'd be interested to see that, but I think it should be a separate issue/thread. It's:

1. Not related to the features in this thread
2. Very specific to a particular controller, whereas the motion sensor stuff applies to all but one platform's controller

@Calinou What are the next steps for motion control support in Godot to move forward? The standards are simple (vector for angular velocity, vector for acceleration), motion controls have had something of a boom in the last year (Fortnite, Call of Duty, CS:GO, Deathloop, Neon White, and more), and even before developers get licenced for a particular platform it'd be great to be able to prototype their controls properly in Godot

Edit: Also in principle I think any 3D engine should make it easy to offer good standard first person camera control out of the box, and the standard of "good" for controller has changed. No need to wait for other engines to catch up when games and input remappers are setting a clear standard imho

[Calinou]

How about adding support for Adaptive Triggers in PS5 Controller/DualSense, since it's also one of the modern input and feedack offered.

Doesn't #1867 cover adaptive trigger support? As I understand it, it's done by sending audio signals to the controller.

What are the next steps for motion control support in Godot to move forward?

4.0 is in feature freeze, so any effort in this direction will be for 4.1 at the earliest.

[mubinulhaque]

I genuinely have looked into both SDL and JoyShockLibrary but it's kinda difficult to implement right now in GDExtension's beta. @JibbSmart feel free to correct me but the documentation is just not that beginner-friendly for SDL and anyone looking to implement gyro controls even as an extension is unlikely to use it.

In order to implement SDL2 as an extension, @JibbSmart the main questions are:

1. How do we know which gyroscopes and accelerometers belong to which controllers?
2. What function allows us to poll (or assign to an event/signal) motion data from a specific controller?
3. How do we calibrate (both manually and automatically) the gyro/accelerometers?

With these answered, I can try to whip up something in C++ or C#. C# is more likely since the support is much further along than C++ but either will be fine.

For anyone else, lurking the thread, GodotSteam currently supports Steam Input and has been recently updated to the latest version of Godot (4.1.1 at the time of writing). This includes gyro and accelerometer support. It's not ideal since it requires you to only distribute to Steam but it's a good starting point until someone eventually comes up with a solution for this proposal.

The Gyro Wiki has provided some incredible work on how gyro should be implemented in any game. Use that as your starting guide.

[protocultor]

@JibbSmart feel free to correct me but the documentation is just not that beginner-friendly for SDL and anyone looking to implement gyro controls even as an extension is unlikely to use it.

I don't agree with that statement.
Check my implementation of gyro aiming for supported controllers (PS4/PS5/Switch/others) on Yamagi Quake 2, which is made in pure C.
The reading of the gyro IMU is done in a single file, sdl.c:
https://github.com/yquake2/yquake2/pull/849/files#diff-cf0de6b02941ef72646ca3c735583a1d349da4a0e0e95558583a5375abe4362e

As this source port already used SDL_PollEvent(&event) to control input events, I only added new logic that controls what it does when receiving SDL_CONTROLLERSENSORUPDATE. That's pretty much it.
I don't do it in this case, but the controller where this event comes from is included in the event itself, so doing something like:
event.cdevice.which == SDL_JoystickInstanceID(SDL_GameControllerGetJoystick(controller))
...should be enough to identify it.
What is included is an example of manual calibration, which is just the sum of all input on every axis on the gyro when calibrating, divide it the amount of input samples, save it to a variable for each axis, and then subtract this variable every time on every SDL_CONTROLLERSENSORUPDATE, for each axis. Works just fine.

[mubinulhaque]

Using GodotSteam and this page as the basis, I did try to build a GDExtension module that binds SDL methods to Godot functions. However, builds currently fail for whatever reason and I think it might be SDL failing. Sadly, the current GDExtension tutorial pages do not provide a lot of help when it comes to adding third-party libraries and binding other DLLs so I don't know where to go from here. I've linked the GitHub page here for anyone that wants to try debugging.

This is the error I get in Scons:
GodotMotionControls_2023-08-21_12-53.txt

I'm using Windows 10, Intel Core i7-4790, NVIDIA GeForce GTX 1650 SUPER and 8 GB of RAM. I was using MINGW to compile.

[NicholasShatokhin]

@mubinulhaque You forgot to link the SDL libraries during build

https://scons.org/doc/0.96.1/HTML/scons-user/x597.html

[mubinulhaque]

I tried that and got a slightly different error.
GodotMotionControls_2023-08-21_15-46.txt

[NicholasShatokhin]

@mubinulhaque Is it yours error?

https://stackoverflow.com/questions/61563619/compiling-sdl2-opengl-glad-project-in-codeblocks-on-windows-undefined-ref

[mubinulhaque]

It is the correct error but I don't know how to add that to SConstruct.

[mubinulhaque]

For those wondering, I am still working on this but I have ran into a major roadblock. Godot requires some libraries that SDL2 needs to be statically linked. Thankfully, it does provide a file that contains what libraries need to be added. Running the command sdl2-config --static-libs nets you this:

-L/usr/lib -lmingw32 -lSDL2main /usr/lib/libSDL2.a -mwindows -Wl,--dynamicbase -Wl,--nxcompat -lm -ldinput8 -ldxguid -ldxerr8 -luser32 -lgdi32 -lwinmm -limm32 -lole32 -loleaut32 -lshell32 -lsetupapi -lversion -luuid

Sadly, I don't know how to add this to the SConstruct file for GDExtension. If anyone knows, do let me know or feel free to try to compile my test module yourself.

[CentiGames]

Are there any news regarding this? I would really love to create a game using gyroscope controls :) (And I somehow can't seem to get Steam Input to work for gyro controls...)

[Calinou]

Are there any news regarding this? I would really love to create a game using gyroscope controls :) (And I somehow can't seem to get Steam Input to work for gyro controls...)

So far, nothing has been decided yet, but Akien mentioned at some point that we should move towards using SDL for input as discussed in #9000. This would make implementing gyro support significantly easier.

[SagaPDev]

I made a GDextention example on Linux with the system installed SDL, i will take a look at @mubinulhaque 's module to see how to make it work on windows.

[SagaPDev]

I made a Godot add-on that allows reading gyro input from game controllers using GDextentions https://github.com/SagaPDev/Godot-SDL-Gyro

[guarapicci]

In the case of linux, in theory, motion sensors should work just fine already, but there's a catch. Two of them, in fact:

• linux pushes out motion sensor data as EV_ABS events . joypad_linux.cpp handles it just fine, but accel, gyro and magnetometer can use up to 9 axes. add in 6 from sticks & triggers and you get 18 axes total. anything beyond the 10th axis will be discarded by the udev backend.
• many controllers have their motion sensors exposed as separate devices. The udev backend ignores anything that that doesn't have both of a button and an axis. The dualshock 4 is one example: udev spawns it as 3 separate devices (controller, motion sensors and touchpad)

I never messed with GDExtensions before, so i don't know if it's even feasible, but maybe an extension can reuse the existing backend and slap some InputEventJoypadIMU event into the Input singleton...

[guarapicci]

After reading up a bunch of different source files, i came up with a plan for a linux implementation. See if these changes align with godot's interests and restrictions:

• (1) add 9 more axis definitions to input_enums.h

enum class JoyAxis {
	INVALID = -1,
	LEFT_X = 0,
	LEFT_Y = 1,
	RIGHT_X = 2,
	RIGHT_Y = 3,
	TRIGGER_LEFT = 4,
	TRIGGER_RIGHT = 5,
	SDL_MAX = 6,
-       MAX = 10, // OpenVR supports up to 5 Joysticks making a total of 10 axes.
+      ACCEL_X = 10
+      ACCEL_Y = 11
+      ACCEL_Z = 12
+      GYRO_X = 13
+      GYRO_Y = 14
+      GYRO_Z = 15
+      COMPASS_X = 16
+      COMPASS_Y = 17
+      COMPASS_Z = 18
+      MAX = 19
};

• (2) make Joypad_linux.cpp expose the motion sensor axes for each controller.

  • (2.1) change Joypad.fd into a collection (array, list, whatever)
  • (2.2) have Joypad_linux.cpp join multiple file descriptors into the same Joypad.
    • group together two or more input file descriptors when their respective files match in VID, PID and the unique identifier (fetched via the EVIOUNIQ ioctl, it's usually a serial number or MAC address). SDL guys figured this out 2 years ago.
    • have udev only discard a joypad if the union of all event types across all file descriptors does not contain the expected amount of buttons and axes.

• (3) change the editor GUI to show the new axes on the action editor event list (if not already generated on-the-fly)

• (4) (optional) built-in calibration and alternative motion sensor data interpretations (player-space, 1£ filter, smoothing, gravity direction vector, etc.)