This OBS plugin allows to use the depth sensor of the Intel RealSense USB cameras to generate a virtual greenscreen. The camera can already be used OBS directly as a normal camera (via V4L2) but this does not use the depth sensor.
With this plugin the camera can be used directly in OBS without OS servoces. The actual access to the camera
and the alignment of the video with the depth field is done by the librealsense2
library. This plugin just performs the masking and it implements the interface to
OBS.
If the required packages are installed, just run make
. The include RPM .spec
file codifies the dependencies which are
- gcc 10 or higher (tested with C++20 only)
librealsense
. Fedora 33 ships the old 2.38 version but it worked for me. Fedora 35 ships 2.40 which also works.- gtkmm 3.
For both librealsense
and gtkmm it is of course necessary to install the -devel
sub-package as well.
To test a librealsense
other than the officially installed version use the
make
command with the variable ALT_LIBREALSENSE
pointing to the appropriate
directory:
make ALT_LIBREALSENSE=/home/drepper/librealsense
This works with the official librealsense
sources off of github.
Before running OBS with the plugin, when all other things can potentially go wrong,
use the testrealsense
binary. It consists of just a few lines of C++ using
gtkmm
to create an appropriately sized window and display the frames the camera
provides. Nothing else. Press the escape key to exit the program. The progam
is not build and shipped when you use the include RPM .spec
file.
After installation the plugin should be immediately found by OBS. Adding it as a source happens just as for other cameras, just select the "RealSense Greenscreen" source.
The property dialog allows to select the device, change the resolution, set the maximum distance (in meters), the size of the depth filter, and greenscreen color.
After the camera source has been added one can use the chroma key filter. To enable
the filter select the RealSense Greenscreen
source in the Sources
list. Right
click on the entry to bring up the context dialog and select the Filters
menu item.
This allows to add the Chroma Key
effect filter. Just make sure to select the
key color as selected for the source and play a bit with the Similarity
and
Key Color Spill Reduction
sliders to get an acceptable result.
Filtering the depth field happens by taking the average value for the last N values. N=1 means no filtering. Any larger value will increase the CPU time and memory needed but it will reduce the effects of the noisy depth field sensor. This sensor frequently produces at random location a non-sensical value and simple averiging seems to be sufficient to take care of this. The default value is four, meaning the average value of the previous four frames is used.
This plugin has so far been tested only on my machine:
- The only camera tested so far is the L515. I hope that the
librealsense2
library handles the other devices the same. - I have not even tested what happens if no camera is attached.
- The frequency of the picture provided is fixed at 30Hz. Not sure whether this needs to be variable.
- The depth sensor is quite noisy. Select an appropriate depth filter size. The larger the size, the more work is required, so restrict the size to a reasonable value for your machine.
- The code allows to set a minimum distance for the mask as well. This setting is so far not carried through to the OBS properties dialog. Not sure this is necessary.
- Obviously, the code has only been tested under Fedora.
Connecting to the camera sometimes fails. One will see then errors
from librealsense
like
RealSense error calling rs2_pipeline_wait_for_frames(pipe:0x1ffabe0):
Frame didn't arrive within 15000
For me this does not mean a problem in the code. Instead, it seems related to the USB device connection. Unplugging the camera and then reattaching it usually fixes this for me.
In future I might try to automatically handle this. Linux allows to reset
an USB device through an ioctl
call with USBDEVFS_RESET
. I will give
this a try the next time I run into the problem.
If the camera is already in use the plugin will not report any resources to OBS. Any already source in a scene will just be zero-sized.
Ulrich Drepper drepper@gmail.com
Parts of the realsense-greenscreen.cc are derived from the rs-align-advanced
example in Intel's librealsense
which has the copyright notice
// Copyright(c) 2017 Intel Corporation. All Rights Reserved.
// License: Apache 2.0. See LICENSE file in root directory.