Samples using different Buffer types causes ClassCastException

[cegredev]

I am working on a video editor and want to record multiple sounds (samples) per frame, because most videos contain sound and you want to add your own ones as well. My approach to solving this was to merge the samples array of all the audio elements together like this:

for (...) {
    Frame videoFrame = grabber.grab(); // Both FFmpegFrameGrabbers
    Frame audioFrame = audioGrabber.grab();

    Buffer[] oldSamples = videoFrame .samples;
    Buffer[] addSamples = audioFrame.samples;
    Buffer[] newSamples = null;
    if (oldSamples != null && addSamples != null) {
	    newSamples = new Buffer[oldSamples.length + addSamples.length];
	    for (int j = 0; j < oldSamples.length; j++) {
	     	    newSamples[j] = oldSamples[j];
	    }

	    for (int j = oldSamples.length; j < newSamples.length; j++) {
		    newSamples[j] = addSamples[j - oldSamples.length];
	    }
     }

    videoFrame.samples = newSamples;
    recorder.record(videoFrame);
}

However, this gives me a ClassCastException at "recorder.record(frame);", saying that it can't cast from "java.nio.DirectShortBufferU" to a "FloatBuffer". While testing I found out, that the error accurs the first time it gets to a sample which came from the audioGrabber, meaning a file that was imported as a mp3 and not alongside a video as mp4. How do I solve this? Or should I tackle my goal differently?

[saudet]

You'll need to make sure that all Buffer are of the same type, for example, either all 16-bit short values, or all float values. We can't have a mix of both.

[cegredev]

Hey, thanks for the quick reply.
How do I do that though? Casting won't work.

[saudet]

One easy way is by calling FFmpegFrameGrabber.setSampleFormat() before start(). For examples, see https://github.com/bytedeco/javacv/blob/master/platform/src/test/java/org/bytedeco/javacv/FrameGrabberTest.java

[cegredev]

I'm sorry, but the error still persists. What exactly shoud I set the sampleFormat to? I tried it with the other grabbers sampleFormat, but that didn't help, The example you provided also seems to be outdated, as it only shows off grabber.setSampleMode and passes a value from a class or enum ("SampleMode") in that doesn't exist or is not accessible to me.

EDIT
I noticed that the sampleFormat and other variables first get set after start() has been called, so I tried setting it after that. Sadly, it still doesn't work, but I can try some more stuff now.

SECOND EDIT
So, uh, apparently you can't change any of a frameGrabbers variables after start() has been called, but anything you set it to beforehand just gets replaced with standard values, basically meaning you can't change it at all... What am I missing?

[saudet]

Ah yes, sorry, that's what setSampleMode() is for, although setSampleFormat() should also work... Let me fix that.

[cegredev]

Okay so I just had some time to test some more, and as it turns out the method setSampleMode() does not exist at all or is not accessable (even when extending a FrameGrabber class). For example this code doesn't work:

final FFmpegFrameGrabber audioGrabber = new FFmpegFrameGrabber(audioFile);
audioGrabber.setSampleMode(0); // doesn't exist (0 just for simplicity)

EDIT
Okay I just took a look at the source for FrameGrabber.java, because I really don't want to waste your time and I too don't understand why I am not allowed to call the method or access the SampleMode enum. Could I have gotten the maven dependency wrong and somehow picked an older version?

[saudet]

It works just fine, take a look at the sample code I gave you above: https://github.com/bytedeco/javacv/blob/master/platform/src/test/java/org/bytedeco/javacv/FrameGrabberTest.java
I can't fix what isn't broken!

[cegredev]

Of course you can't. I just checked, and I truly do have the wrong version in my pom.xml. I can't change anything at the moment, but I imagine that's it.

[cegredev]

Yeah, that was it. Sorry for the trouble. However, the audio is played back much slower than it actually is, causing it to sound weird. Should I open a new issue for this once I feel like I'm done testing, or...?

[saudet]

Of course you can't. I just checked, and I truly do have the wrong version in my pom.xml. I can't change anything at the moment, but I imagine that's it.

Right, setSampleFormat() works just fine, there's nothing to be fixed.

Yeah, that was it. Sorry for the trouble. However, the audio is played back much slower than it actually is, causing it to sound weird. Should I open a new issue for this once I feel like I'm done testing, or...?

You're probably just not setting the sample rate properly. Make sure that you do call setSampleRate() with a value that makes sense!

[cegredev]

Did not notice your reply here, sorry about that.
So I tried setting the sample rate on both grabbers and the recorder to 44100, and it sadly doesn't work. I also tried setting the audioBitrate, the audioChannels, the audioCodec and the sampleFormat of the grabbers to the ones of the recorder.

EDIT
So the only difference between the recorders settings and the grabbers settings that I could spot are the audio bitrates. However, I am setting those values before start(), but the changes seem to be reverted after it is called.

But is this (the code at the top) the correct way of achieving my goal (exporting a video with sound from multiple audio sources) anyways? Not understanding how Buffers work or can be transferred to sound, it's just the first thing I came up with and I can't believe it's actually right.

[saudet]

Setting the sample rate on the grabber might not have any effect, especially for files. You'll need to use for the recorder the sample rate that you get from the grabber.

[cegredev]

The samples rates are the same anyways (even without me setting them). Also the problem with that would be, that if I had multiple sources with different sample rates, setting the recorder's sample rate wouldn't work.

[saudet]

If you need that, start looking at FFmpegFrameFilter to do some resampling.

[cegredev]

Okay, this is what I got so far:
Creation of the filter:

final FFmpegFrameFilter audioFilter = new FFmpegFrameFilter("aresample=44100", recorder.getAudioChannels());
audioFilter.start();

However, this throws the following exception:

org.bytedeco.javacv.FrameFilter$Exception: av_buffersrc_add_frame_flags() error -22: Error while feeding the filtergraph.

This is how I intend to use the filter (modification to the code above):

audioFilter.push(audioGrabber.grab());
Buffer[] oldSamples = audioFilter.pull().samples;

Will this work (and how do I get rid of that exception)?

[saudet]

Actually FFmpegFrameGrabber.setSampleRate() should work. Could you provide an example that doesn't work?

[cegredev]

Files.zip

Of course. I decided to leave every bit of code in, because after having an error in the pom.xml, this one could be anywhere.

import java.io.File;
import java.nio.Buffer;

import org.bytedeco.javacv.FFmpegFrameGrabber;
import org.bytedeco.javacv.FFmpegFrameRecorder;
import org.bytedeco.javacv.Frame;
import org.bytedeco.javacv.FrameGrabber.Exception;

public class RecorderTest2 {

	public static void mergeVideoAndAudio(final File videoFile, final File audioFile)
			throws Exception, org.bytedeco.javacv.FrameRecorder.Exception, org.bytedeco.javacv.FrameFilter.Exception {
		final File output = new File(System.getProperty("user.home") + "\\desktop\\test.mp4");
		System.out.println(System.getProperty("user.home") + "\\desktop\\test.mp4");

		final FFmpegFrameRecorder recorder = new FFmpegFrameRecorder(output, 1920, 1080, 2);
		recorder.setFormat("mp4");
		recorder.setSampleRate(44100);

		final FFmpegFrameGrabber videoGrabber = new FFmpegFrameGrabber(videoFile);
		final FFmpegFrameGrabber audioGrabber = new FFmpegFrameGrabber(audioFile);

		videoGrabber.setSampleMode(FFmpegFrameGrabber.SampleMode.FLOAT);
		audioGrabber.setSampleMode(FFmpegFrameGrabber.SampleMode.FLOAT);
		videoGrabber.setSampleRate(44100);
		audioGrabber.setSampleRate(44100);
		videoGrabber.setSampleFormat(recorder.getSampleFormat());
		audioGrabber.setSampleFormat(recorder.getSampleFormat());

		videoGrabber.start();
		audioGrabber.start();
		recorder.start();

		for (int i = 0; i < 300; i++) {
			final Frame frame = videoGrabber.grab();
			
			Buffer[] oldSamples = frame.samples;
			if (oldSamples == null)
				oldSamples = new Buffer[0];
			
			Buffer[] addSamples = audioGrabber.grab().samples;
			if (addSamples == null)
				addSamples = new Buffer[0];
			
			Buffer[] newSamples = new Buffer[oldSamples.length + addSamples.length];
			
			for (int j = 0; j < oldSamples.length; j++) {
				newSamples[j] = oldSamples[j];
			}
			for (int j = oldSamples.length; j < newSamples.length; j++) {
				newSamples[j] = addSamples[j - oldSamples.length];
			}
			
			frame.samples = newSamples;
			if (newSamples.length <= 0)
				frame.samples = null;
			
			recorder.record(frame);
		}

		recorder.stop();
		recorder.release();
		recorder.close();
		
		videoGrabber.stop();
		videoGrabber.close();
		audioGrabber.stop();
		audioGrabber.close();
	}

	public static void main(String[] args)
			throws Exception, org.bytedeco.javacv.FrameRecorder.Exception, org.bytedeco.javacv.FrameFilter.Exception {
		
		final File videoFile = new File("test\\video_test.mp4");
		final File audioFile = new File("test\\test_audio.wav");
		
		mergeVideoAndAudio(videoFile, audioFile);
	}
}

The output and input file(s) as well as the pom.xml are attached.

[saudet]

Could you simplify that to only 1 file?

[cegredev]

I'm not sure if I understand what you mean, but I guess you could remove the audioGrabber and see what happens in that case like this:

public static void mergeVideoAndAudio(final File videoFile)
			throws Exception, org.bytedeco.javacv.FrameRecorder.Exception, org.bytedeco.javacv.FrameFilter.Exception {
	final File output = new File(System.getProperty("user.home") + "\\desktop\\test.mp4");

	final FFmpegFrameRecorder recorder = new FFmpegFrameRecorder(output, 1920, 1080, 2);
	recorder.setFormat("mp4");

	final FFmpegFrameGrabber videoGrabber = new FFmpegFrameGrabber(videoFile);
	videoGrabber.setSampleMode(FFmpegFrameGrabber.SampleMode.FLOAT);
	videoGrabber.setSampleRate(44100);
	videoGrabber.setSampleFormat(recorder.getSampleFormat());

	videoGrabber.start();
	recorder.start();

	for (int i = 0; i < 300; i++) {
		recorder.record(videoGrabber.grab());
	}

	recorder.stop();
	recorder.release();
	recorder.close();
		
	videoGrabber.stop();
	videoGrabber.close();
}

Result: The audio of the video plays back, however with a bit of latency.
If you were to replace the samples of the frame with the samples of the audioGrabber, the song would play back, but a little slower than intended, since the finished video is then 6 seconds long, although it should be 5 for 300 frames at 60 fps.

Keep in mind though that the kind of hard-coded code in this and the previous post is just for testing purposes, and will later be replaced with code that can handle a variable amount of files.

[saudet]

Ok, so the problem is with "latency"? That just means you're missing frames. Make sure that the audio frames and the video frames you write span the same exact amount of time. See issue #1333 for a discussion about that.

/cc @anotherche

[anotherche]

I'm not sure I understood what the original goal was. From the first message, it seems that @SchredDev would like to add an additional audio track to the file. But the code looks so that new portions of sound are sequentially added to existing sound buffers. It seems to me, whatever the original goal, such a method will not give a normal result.
Regarding the possibility of adding an audio track (so that later you can select a track during playback) - it seems that the current version of the FFmpegFrameRecorder allows you to create only one video stream and one audio stream in a file (see startUnsafe() code). So the recorder code needs to be reworked to add this feature (ffmpeg allows this). By the way, the FFmpegFrameGrabber in the current version also opens only one video and one audio stream from files (it just looks the first ones from the list of corresponding streams, which is also seen from its startUsafe() function).

[cegredev]

Okay, I'm just going to try to explain my goal in detail.

The editor I am working on creates a BufferedImage for each frame in the movie. This is were I am going to get all the video footage from when exporting (the code above). This means that there won't be a video-file in this method, because that data will be provided directly as a frame. At the moment it is just in there so I can get easy access to visual and more audio data (because it provides both).
I do however need a way to create videos alongside some audio. That's why I have to somehow merge the audio of multiple files and record it.

so the problem is with "latency"?

Not exactly. I am guessing the audio is played back slower, even when there's only one file, because that's what happens when I merge multiple files (just more extrem) and the video is longer than it should be. That would cause the audio to be desynchronized.

[anotherche]

@SchredDev, It’s still not clear what “merge” means. Is there a need for the resulting video to have several different audio tracks, or for sound from many sources to be mixed into one stream so that everything sounds together?

[cegredev]

The later one.

[anotherche]

Maybe I don’t know something about the possibilities of ffmpeg or JavaCV, but it seems to me that just appending extra samples to the end of existing buffer can not lead to a normal result, including mixing of two sound sources. Theoretically, mixing assumes resampling (decoding to uncompressed format, summation, encoding) which should include the following operations: 1. portions of the same duration are taken from two sound sources; 2. they are decoded in PCM format with the same bitdepth and sample rate; 3 resulting samples are scaled (to prevent possible clipping) and summed (old sample + new sample, sample by sample). 4 result is recorded (encoded).
In the ffmpeg cli, this feature is implemented as an amix filter. I don’t know if FFmpegFrameFilter can do the same. Maybe @saudet can tell.
A simple appending of samples, as in your code, leads in my opinion to the following. Sound in the video is cut into short frames of a few hundredth of second. When a sound card plays decoded samples from these frames, you get the usual continuous sound. If additional samples of approximately the same duration are attached to each sound frame, then you get sound frames in the form (old + new) + (old + new) + ... Sound card will play them as they come. Now the duration of the sound of each frame has increased approximately twice. The overall result will sound twice as long, and therefore, it will seem that the sound has become slowed down two times. The human brain will hear it as if two slowed down sounds are superimposed. But, they will sound somewhat strange, since the pieces of each sound go intermittently - they stop and resume every few hundredths of a second. This is exactly what you hear, as I guess.

[saudet]

As far as I understand, he's just trying to add audio channels, but he'll need the same amount of samples for all channels to do it that way, or they won't be synchronized.

[cegredev]

@anotherche Seems like that's what could be happening here! I've uploaded a video showcasing the expected and actual outputs.

If it is, could you please point me to an example showcasing what you just explained?

[saudet]

If you're looking into mixing audio samples, the amix filter wold be the easiest thing to use:
https://ffmpeg.org/ffmpeg-filters.html#amix

[cegredev]

Okay, thank you.

[cegredev]

Okay so judging by the examples I cound find, I'd have to initialize the filter like this, right?
final FFmpegFrameFilter filter = new FFmpegFrameFilter("amix=inputs=2:duration=first:dropout_transition=3 \" + outputPath + "\"", 2);

But how would I set the inputs in that case?

[saudet]

See issue #1214 about that.

[cegredev]

Okay, I copied the code they showed in their issue, and I don't get an error anymore (when calling start). However, I still do not get how to give the filter the inputs and get the output. This is what I tried:

filter.pushSamples(frame1.samples.length, frame1.audioChannels, frame1.sampleRate, videoGrabber.getSampleFormat(), frame1.samples);
filter.pushSamples(frame2.samples.length, frame2.audioChannels, frame2.sampleRate, audioGrabber.getSampleFormat(), frame2.samples);
frame1.samples = filter.pullSamples().samples;
recorder.record(frame1);

which throws the error:

org.bytedeco.javacv.FrameFilter$Exception: av_buffersrc_add_frame_flags() error -22: Error while feeding the filtergraph.
at org.bytedeco.javacv.FFmpegFrameFilter.pushSamples(FFmpegFrameFilter.java:622)

at the first line.

[saudet]

Check the log to get more information about that error.

[cegredev]

Oh. My mistake, didn't see that. It says:

[1:a @ 000002408b2ce5c0] filter context - fmt: s16 r: 44100 layout: 4 ch: 1, incoming frame - fmt: flt r: 44100 layout: 3 ch: 2 pts_time: NOPTS
[1:a @ 000002408b2ce5c0] Changing audio frame properties on the fly is not supported.

I found this and that issue concerning the second output, but those don't seem to be the case here.

[saudet]

Make sure to call FFmpegFrameFilter.setAudioChannels(), FFmpegFrameFilter.setSampleFormat() and FFmpegFrameFilter.setSampleRate() with these values before calling FFmpegFrameFilter.start()

[cegredev]

Thanks, the error is gone now. However, pull() and pullSamples() of the filter jut return null, and pullImage() even causes a crash outside the Java VM. But that's probably only because I'm not feeding the filter any visual data. Why are the other ones returning null though?

final Frame frame1 = videoGrabber.grab();
final Frame frame2 = audioGrabber.grab();

if (frame1 != null && frame1.samples != null) //Returns true when appropriate
	filter.pushSamples(frame1.samples.length, frame1.audioChannels, frame1.sampleRate, videoGrabber.getSampleFormat(), frame1.samples);

if (frame2 != null && frame2.samples != null) //Returns true when appropriate
	filter.pushSamples(frame2.samples.length, frame2.audioChannels, frame2.sampleRate, audioGrabber.getSampleFormat(), frame2.samples);

Frame audioFrame = filter.pull(); //Always returns null
if (audioFrame != null) //Always returns false
	frame1.samples = audioFrame.samples;

recorder.record(frame1);

[saudet]

Don't call pushSamples(), call push().

[cegredev]

It finally works, thank you so much! I used push() on frame1, pushSamples() on frame2 and pullSamples() on the filter.