Webcam streaming issue

[Overlord999]

Hi,

I am using the sample programs to test webcam interface. I have encountered some issues summarizing below. I am testing with two webcams and the phonecam.

Using MultipleCameraExampleOpenCvAlongsideVuforia:
screen 1 (Vuforia) screen 2 (EasyOpenCV)
webcam 1 (ok) phonecam (ok)
webcam 1 (ok) webcam 2 (blue screen)
phonecam (ok) webcam 2 (ok)

Using MultipleCameraExample:
screen 1 (EasyOpenCV) screen 2 (EasyOpenCV)
webcam 1 (ok) phonecam (ok)
phonecam (ok) webcam 1 (ok)
webcam 2 (ok) webcam 1 (blue screen)
webcam 1 (ok) webcam 2 (blue screen)

I also observed that the phone would die after I ran the program a few times. I then use the MultipleCameraExample.java but enable one camera only. I put StartStreaming() and StopStreaming() in a loop with delays in between. If I used the phonecam, the loop could run forever until I stopped it. However, when I used a webcam, the loop would stop after 4 iterations.

[Windwoes]

I also observed that the phone would die after I ran the program a few times. I then use the MultipleCameraExample.java but enable one camera only. I put StartStreaming() and StopStreaming() in a loop with delays in between. If I used the phonecam, the loop could run forever until I stopped it. However, when I used a webcam, the loop would stop after 4 iterations.

This is a bug in the FTC SDK, not related to EasyOpenCV at all. ftctechnh/ftc_app#681. There is not much you can do about it. However, you can avoid the issue by unplugging and replugging the phone after running it 3 times.

---

Now, regarding your table of test results... For reference, what the blue screen means is that the viewport was successfully added to the UI and the viewport rendering thread is alive and kicking, simply waiting for its first frame from the camera hardware. I feel like the issues you saw initializing a webcam may be related to the bug I mentioned about, but I'm not sure. Can you re-test each situation, but with unplugging and replugging the USB from the phone before each test? If it still has problems, can you provide the logs from the failures?

[Overlord999]

The test result table was compiled with phone and robot in fresh start. Otherwise, there could be all kind of weird error messages. What kind of log are you looking for?

[Windwoes]

I'm looking for the log from the robot controller app (robotControllerLog.txt on the phone's internal storage). If you could delete it before reproducing the issue so as to remove any unrelated previous logging that would be good.

[Windwoes]

Also, what robot controller phone are you using?

[Overlord999]

Here is the log file. This was the case with two webcams using MultipleCameraExample. I used two webcams. One was showing a blue screen. We were using Samsung Galaxy S5 with the same issue. We switched to Moto G5+ with the same issue. Upgraded the G5+ to Oreo 8.1 still with the same issue.

robotControllerLog[1].txt

[Windwoes]

Are you using a powered USB hub?

[Overlord999]

Yes. I am using a powered USB hub.
Another test I did was to alternate the two webcams in a loop.
loop {
start streaming webcam1
stop streaming webcam1
start streaming webcam2
stop streaming webcam2
}
It worked for two complete cycles, and then the program died.

[Windwoes]

Yes. I am using a powered USB hub.

Hmm. I tested just now using a powered USB hub with 2 Logitech C920 webcams and a Pixel XL phone and it ran just fine.

Another test I did was to alternate the two webcams in a loop.
loop {
start streaming webcam1
stop streaming webcam1
start streaming webcam2
stop streaming webcam2
}
It worked for two complete cycles, and then the program died.

This is caused by the bug in the FTC SDK, as I mentioned before. This will happen even with the SDK's Vuforia webcam sample. Run it too many times in a row and it will die.

[Windwoes]

Update: I was able to reproduce the issue. I took a look at your log and noticed these lines:

D/Uvc: [stream.cpp:931] ...uvc_stream_start_bandwidth() rc=-1
E/UvcStreamHandle: [jni_streamhandle.cpp:155] failed to start streaming: uvc_stream_start()=-1(UVC_ERROR_IO)

This seems to indicate that there wasn't enough bandwidth available on the USB bus to start the stream from the second camera. On a hunch, I tried changing my program to stream at 640x480 from both webcams, and I was able to reproduce the issue. However, streaming at 320x240 from both webcams (which is what I tested before) works just fine.

[Overlord999]

It works for me at 320x240 too. Thank you very much for solving the problem for us.

If we only need to snap some still pictures from time to time. Is there any easy way to get higher resolution pictures without loading down the bus and CPU by streaming?

[Windwoes]

We can calculate how many Mbps of bandwidth will be required for a stream with

(FRAME_WIDTH * FRAME_HEIGHT * BITS_PER_PIXEL * FRAMES_PER_SECOND) / 1000000

The SDK currently only supports streaming in YUY2 format (which is 16 bits per pixel) and EasyOpenCV currently only supports selecting an (up to) 30FPS stream from a webcam, so the equation becomes:

(FRAME_WIDTH * FRAME_HEIGHT * 16 * 30) / 1000000

We can then see then, that:

YUY2 320x180 @ 30FPS stream needs up to  27.648Mbps of bandwidth
YUY2 320x240 @ 30FPS stream needs up to  36.864Mbps of bandwidth
YUY2 640x480 @ 30FPS stream needs up to 147.456Mbps of bandwidth

Two 640x480 streams, then, would need up to 294.912Mbps of bandwidth. The USB 2.0 bus can theoretically handle up to 480Mbps, but if you throw in an Expansion Hub which also takes some bandwidth, as well as account for overhead of the USB protocol itself, and the fact that the USB controllers in the phones probably aren't the greatest, then it does make sense how nearly 300Mbps might be too much for it to handle.

[Windwoes]

If we only need to snap some still pictures from time to time. Is there any easy way to get higher resolution pictures without loading down the bus and CPU by streaming?

I can look into supporting higher resolution streams at lower frame rates, but I'm not sure implementing single-shot pictures would be possible without modifying the SDK's UVC driver, for which we do not have the source code. Integrating a 3rd party UVC driver would solve lots of problems, but the amount of work required to make that happen is not appealing....