Add procedural and parametrizable audio support

[reduz]

Describe the project you are working on

Godot

Describe the problem or limitation you are having in your project

Sound designers in the industry end up having to use tools such as FMOD or Wise, with not really any FOSS alternative. Godot users are forced to use this bit of proprietary software in order to have more advanced audio in their games.

Interactive music is being handled by a separate PR/proposal being worked on, so this focuses entirely on procedural audio.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

This would be a special AudioStream resource in Godot that would make it possible to have procedural/parametrizable audio playback in Godot.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

A new AudioStream class will be added: AudioStreamGraph. When edited, it will look like this:

The general idea is that the editor will be divided in two.

• Audio Graph
• Stream Positions

In the audio graph, several AudioGraphNodes will exist. Those have input and output Audio connections, as well as Controller connections (as a different type of data port).

The following audio graph nodes will be present:

Basic

• Output (comes by default)
• Stream: Allows setting any AudioStream in Godot. When one of these audio graph nodes is present, user will be able to adjust the offset on where it starts playing.
• Effect: Allows setting any AudioEffect from Godot (from the existing ones), which can change the sound accordingly. Audio effects export parameters, so these parameters will appear as input controller connections.
• Parameter: A named parameter, this parameter exports a controller output and can be named. This will appear as editable property in the AudioStreamPlayer[2D/3D] nodes. If a parameter is not connected to a controller, a default value can be set in the input port itself, like with Visual Shader and Visual Script.

Generators

These generate audio output:

• Oscillator: Simple oscillator (sine/saw/etc)
• Noise: Noise generator (pink/white/blue/etc).

Operators

These combine audio outputs:

• Mixer: Adds two audio streams
• Modulator: Multiplies two audio streams
• Frequency Modulator: Does frequency modulation to two audio streams

And the idea is to have what is common in this type of software for controlling parameters, such as LFO, Envelopes, etc. This is just the general proposal.

If this enhancement will not be used often, can it be worked around with a few lines of script?

Audio is low level.

Is there a reason why this should be core and not an add-on in the asset library?

While this could be an add-on, games often require a mature and maintained solution for this. Given the popularity of tools such as Wise and FMOD, it sounds like it should be better this is core.

[MarioLiebisch]

I'm totally for this. Once planned to try something similar as an addon for chiptunes specifically, but I've never found the time to actually do so (so far).

First thought about the proposal above: I don't think this would need a timeline (nor should it have one). Instead there should probably be a Delay modulator node or something like that (along with others to cut parts of streams, too).

But just to put this in for discussion – random idea – might be worth maybe even making this fully scriptable (think VisualShader vs. Shader). The big thing would be the framework either way, considering most processing can be broken down to rather simple math operations.

Quick example for a sinus generator:

audio_node generator; // has no input sample

input float frequency = 10000.0f;
input float amplitude = 1.0f;

// Basically called once per requested sample
void modulate() {
  SAMPLE.lr = amplitude * sin(TIME * 2 * PI * frequency);
}

Or some stereo balance modulator:

audio_node modulator; // gets a sample as input

input float balance = 0.5f; // 0.0 = full left, 1.0 = full right

void modulate() {
  sample s = mono(SAMPLE); // reduce to mono
  SAMPLE.l = (1.0f - balance) * s;
  SAMPLE.r = balance * s;
}

So one script would define a single node (input or modulator), but full scripts could also be merged to single "complex" nodes again. Maybe overthinking, but I'd consider this very neat (and also very useful in education/audio processing) and maybe something for GPU processing, too (as long as no samples are based on previous ones)?

[theraot]

I would like to point out that we have AudioStreamGenerator/AudioStreamGeneratorPlayback. Sadly, getting the data from a stream into that is… ern… well, there was an issue about that, but never a proposal: AudioStreamSample.data compatibility with AudioStreamGeneratorPlayback - Without that, it is not easy to implement the addon, or to do this from code for a game.

[alexfreyre]

Maybe PureData (foss) and MaxMSP can give some ideas. They both are used to create interactive projects with procedural sounds/music/synthesizers.

I have seen another software that target procedural music creation for live events either via nodes or via code. It's called PraxisLive (foss). Also, you can make visual effects or shaders on it but it is made by a musician-programmer.

[starry-abyss]

Also what do you think about this FOSS project: https://github.com/soul-lang/SOUL? Perhaps, it could be leveraged as the "audio shader" language

[michaelhartung]

This is awesome :)!

Some thoughts from my perspective as Technical Game Sound Designer who's worked on a number of Unity/UE4 using both their native audio and FMOD/Wwise (and one Godot project so far :)):

One of the key features that makes FMOD / Wwise so attractive, apart from from all the convenient mixing and helpful profiling tools is that they're event driven. An event can do a plethora of things ranging from simple audio playback of a simple random wave container to setting global parameters that affect many currently playing sounds, stop other sounds etc.. The designers are given a lot of freedom while the code-side requirements are fairly non-complex most of the time.

Where "procedural" or parameter driven audio gets very interesting in a game audio context, is sounds affecting other sounds, when the system can talk to itself, update state, and talk back to the game loop so gameplay can react to sound instead of the other way around.

The one new native audio implementation that I'm most excited about currently is UE5's new meta sounds:

https://docs.unrealengine.com/5.0/en-US/AudioFeatures/MetaSounds/MetasoundsReferenceGuide/

Maybe there's some inspiration / ideas to be found there that could fit the Godot way of doing things.

[reduz]

@MichaelKlier I have seen some of the work that Epic is doing on MetaSound, but I have the feeling it is largely too complex for a musican or sound designer to use. For something so complex, I think it would be easier to just use PureData or similar as an add-on. I wanted to focus on having a simpler workflow where you can still do most of the most common audio manipulation tasks.

[gfazzito]

Strongly agree. Implementing an interface and a workflow that musicians and sound designers are used to (for example: a DAW like Reaper and a middleware like Wwise), would be an excellent improvement.

[bughandler]

MetaSound is too complex to use, it would be good enough for Godot to have a solution that is capable to replace FMOD/Wwise

[michaelhartung]

@reduz totally agree that Metasound is certainly complex. I didn't mean to imply to do sth. similar, merely suggesting to take inspiration/ideas by some of the functionality it implements :). Metasound is the only native audio implementation that I would personally consider to use over 3rd party Middleware (native UE4 is capable as well but still limited). Again, always considering the actual requirements of the project, not every project needs middleware.

I guess it really depends on what the end goal is. If it's about getting up to par with audio middleware out of the box to make it an attractive solution for sound designers to use, then it probably needs to offer at least the core set of features these tools have.

On that note, they only have very basic implementations of actual sound generators (sinewaves etc.), with Wwise having some more advanced ones for certain types of ambient sounds like wind and I don't know many people who are using them. And procedurally generating sounds like. wind/fire etc. using just sound generators paired with filter nodes, envelopes, lfo's etc. is a complex task on it's own.

By planning to offer this kind of functionality, when I read the outline correctly, you're kind of entering the PD/MaxMsp/Metasounds realm anyway.

I believe this discussion could benefit from actively bringing in more voices of audio people who are working with these tools on a daily basis.

[MarioLiebisch]

Regarding sound generators: I wouldn't expect Godot to have any complex generators, but certainly think it would be nice to have some very simple out of the box solution to generate basic effects or sounds similar to what bfxr and similar tools can do.

[SoyoTamo]

Without a doubt this would be something that I would use in all my projects. BFXR is ideal for small or rapid development projects, seeing something like this integrated into the engine together with the initial proposal would leave all the basics served with respect to audio, getting closer and closer to having to leave GODOT to do something, so less something simple.

[GeorgeS2019]

Topic: AudioStreamSample.data compatibility with AudioStreamGeneratorPlayback

Just curious, has the issue proposed here being addressed?

do some bit-shifting as well as a signed conversion for 16-bit samples

for interoperability between AudioStreamSample.data and AudioStreamGeneratorPlayback?

[TokageItLab]

Recently, CLAP an audio plugin API, was released by bitwig (modern DAW software developer), and u-he (long-established synthesizer developer).

https://github.com/free-audio/clap

It is MIT licensed, unlike VST, and appears to be compatible with Godot license.

Although it depends on how popular the CLAP format becomes, I think it has the potential to cover most procedural audio use cases by adding MIDI Input Node and CLAP Plugin Node to AudioGraph.

[gregcsokas]

there is a Glicol named madness, it basically says
"make music with code. connect different nodes like a synth, sequencer or sampler."
it's written in Rust
https://glicol.org/

It's pretty fun stuff, and the language that uses is simple :)

[nobuyukinyuu]

@theraot wrote:

I would like to point out that we have AudioStreamGenerator/AudioStreamGeneratorPlayback. Sadly, getting the data from a stream into that is… ern… well, there was an issue about that, but never a proposal: AudioStreamSample.data compatibility with AudioStreamGeneratorPlayback - Without that, it is not easy to implement the addon, or to do this from code for a game.

@GeorgeS2019 wrote:

Just curious, has the issue proposed here being addressed?

I never made a new proposal for it, because while it would be a convenient function, it can be implemented in code (albeit with a small bit of hassle). One thing I am considering proposing though is to request that AudioStreamGeneratorPlayback have some more input types in C# because for whatever reason all my projects start pegging the audio thread in Godot pretty quickly and cause the project to drop frames/become unresponsive, and I suspect it has something to do with having to feed it a new buffer (or copy an existing buffer) every time. That would make the feature less than ideal for an all-purpose solution.

Haven't looked into what happens if you feed more frames than it wants or if this can buy headspace so the same size buffer can be reused over and over again (just skip enough process frames), but maybe at least being able to take an ArraySegment<Vector2> could mitigate the issue and prevent an array copy every process frame... (Edit: Honestly I can't really be sure what's going on under the hood anyway since c# code in the Godot runtime doesn't seem like it can easily be profiled, at least at my level of knowledge)

[FlashboxSound]

Is this project still in the works somehow? I find it extremely interesting and I think it would add a unique edge to Godot over other engines. If not implement something from scratch maybe a very reliable way to work with Pure Data (also FOSS)?
/ Curious new Godot user

[alexfreyre]

Sharing some ideas about this PR, SuperCollider (foss) is something amazing and does what this PR is pointing to, although it's only through code. Maybe its the server and language could be used within Godot as it is intended to be used for interactivity and real-time.

[dromer]

I would love to add support for Godot in the Heavy compiler (Pure Data to C/C++ conversion): https://github.com/Wasted-Audio/hvcc
Any directions or help with getting this done is appreciated!

We currently have support for Unity and WWISE targets, but it would be great to have a full opensource stack and better integration with Godot.

If Godot can provide its own UX-design aspects and multi-channel (and spatial) audio handling and load audio plugins this would be a nice separation of concerns. Heavy won't be able to run your entire audio suite, but handle specific procedural DSP tasks. Both audio generation ("instrument") and processing ("effects") types are at minimum useful.

BTW it was rumored that Unreal would move to implementing CLAP audio plugin support. That could actually be an interesting move to go for CLAP hosting as the main interface for Godot audio plugin devs. We currently already have basic CLAP support in Heavy (which could save on some technical debt, compared to designing+implementing a spec from scratch).

[IntangibleMatter]

Commenting so that I get notified of updates on this, and also to note my high level of investment in this proposal.

[Calinou]

Commenting so that I get notified of updates on this

You can use the button in an issue's sidebar to subscribe to an issue without leaving a comment:

[IntangibleMatter]

Now that godotengine/godot#64488 is approaching completion, I think it would make sense to start discussing this again, how it works with the new system, and whether the graph approach is still best.

[starry-abyss]

@IntangibleMatter Well, that PR implements adaptive music, this proposal is about building synths and mixers. Most of the features are not overlapping I think.

[starry-abyss]

Perhaps allowing sends between buses (like in most DAWs) would be enough for the graph functionality or at least as a quick starting point.

[afarra6]

I feel it would be beneficial to add more synthesis options than the listed oscillators/noise generators. Having a Karplus-strong, granular, wavetable, or other options with some sort of midi control might be a good starting place. A stream could be structured similarly to the randomization stream (in terms of holding multiple different streams) and each assigned a midi-voice. So something like this:

• AudioStreamPlayer
• • BandStream (Takes in all MIDI data and routes it to substreams)
• • • Instrument 1 (Sine wave) (MIDI voice 1)
• • • Instrument 2 (Karplus-strong) (MIDI voice 2)
• • • Instrument 3 (Pink-noise generator) (MIDI voice 3)

or just using the AudioStreamSynchronized found here: godotengine#64488

(Though if the intent is to be able to send MIDI data to the substreams I'm not sure if that's the right approach)

I mention MIDI only because Godot currently does not officially support OSC, as I believe OSC would be a much preferable solution for stream communication than MIDI.

The instrument streams could be a single stream resource that uses something like the synthesis toolkit (which the ChucK language uses) and then use an enum to choose the "instrument". That way there would not be a separate stream for each oscillator/synthesis/sound generator type, just a single InstrumentStream.

I think the graph approach will be the most approachable for those who have worked with something like MaxMSP or pd in the past, so I think it's a useful feature to keep. Though, having some sort of hybrid timeline/graph system may be most useful, sort of like openmusic's maquette system, where the sound generating graph can be inside a larger horizontal block in a timeline. Sort of like this:

[dmrokan]

Implemented a simple tone and noise generator. It uses audio filter backend and UI. These are spectral domain images of the signals produced by this generator.

Sine and saw tooth signals at 400Hz:

White, brown and pink noise spectrum:

Pink noise is based on this article:
https://www2.umbc.edu/photonics/Menyuk/Phase-Noise/kasdin_ProcIEEE_950501n.pdf

I tuned it by trial-and-error for 44100Hz sample rate audio. I think it needs a higher order filter and parameter tuning. I am not sure how it will behave for other sampling rates.

if it is considered as an alternative solution, I am planning to create a PR.

Background information

Tone

Tone is generated by simulating an ideal oscillator. Its goal is generate a signal in this form

$$y(t) = e^{-at} \cos(\omega t + \psi)$$

where $t$ is continuous time, $\omega$ frequency, $\psi$ phase and $a$ is damping. A larger $a$ causes signal quickly vanish. Since we are simulating it, it should be transformed to discrete time domain. It becomes

$$y_n = r^n \cos[\omega n + \psi]$$

where $n$ is an index starting from initial time index $0$, frequency $\omega$ is now normalized by audio sampling frequency and $r$ is a number from 0 to 1. If $r$ is less than 1, signal will fade by time. This signal can be generated by simulating the behavior of an oscillator which is a filter with 2 states ($p$ and $q$) and can be written in this iterative form,

$$p_{n+1} = c_1 p_n + c_2 q_n$$

$$q_{n+1} = p_n$$

starting from an initial $p_0$ and $q_0$. When parameters $c_1$, $c_2$, $p_0$ and $q_0$ are chosen appropriately $p_n$ will be

$$p_n = r^n \cos[\omega n + \psi]$$

Noise

White noise can be generated by Godot's math library's randfn function. Left most column of the second image above proves it. It is constant over all spectrum as expected.

Brown can be generated by accumulating randfn. I think its spectrum in the image above is very close the one here.

Pink is tricky. It is generated by passing the output of randfn from a 10th order FIR filter. The filter coefficients are calculated by using the Equation 104 in the linked article above. If filter order is increased it will behave more like a pink noise generator. There are possibly better methods to generate it, but it needs research.