[Android] When VideoCapture.read is executed, if the stride is longer than the width of the video, an error will occur.

[Dev-Czle]

System Information

OpenCV Version : 4.9.0
Operating System / Platform: Android 14 / Samsung Galaxy Flip3 and Google Pixel 7
Compiler & compiler version: Android Studio

Detailed description

in Android version 14, i use

public boolean read(Mat image) {
        return read_0(nativeObj, image.nativeObj);
}

Previously, we were using OpenCV 4.5.5. After updating the Android version to 14 and OpenCV to 4.9.0, we discovered crashes occurring on some devices with certain videos. The logs for these videos revealed a common issue: the stride reported in the NativeCodec log was longer than the width.
Perhaps it is related to "#22214"

I think the value of the stride is not being initialized when it changes.

 D  c2 config diff is   c2::u32 raw.color.matrix = 3
      c2::u32 raw.color.primaries = 3
      c2::u32 raw.color.range = 1
      c2::u32 raw.color.transfer = 3
      c2::u32 raw.crop.height = 1024
      c2::u32 raw.crop.left = 0
      c2::u32 raw.crop.top = 0
      c2::u32 raw.crop.width = 1280
 D  [c2.qti.avc.decoder#870:2D-BB-Output] popFromStashAndRegister: at 0us, output format changed to AMessage(what = 0x00000000) = {
      int32_t android._color-format = 2135033992
      int32_t android._video-scaling = 1
      int32_t android._dataspace = 257
      int32_t color-standard = 2
      int32_t color-range = 1
      int32_t color-transfer = 3
      int32_t sar-height = 1
      int32_t rotation-degrees = 90
      Buffer hdr-static-info = {
        00000000:  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................
        00000010:  00 00 00 00 00 00 00 00  00                       .........
      }
      int32_t sar-width = 1
      Rect crop(0, 0, 1279, 1023)
      int32_t width = 1280
      int32_t feature-secure-playback = 0
      int32_t frame-rate = 30
      int32_t height = 1024
      int32_t max-height = 4320
      int32_t max-width = 8192
      string mime = "video/raw"
      int32_t priority = 0
      int32_t vendor.qti-ext-dec-timestamp-reorder.value = 1
      int32_t color-format = 21
    }
 D  [c2.qti.avc.decoder#870:2D-BB-Output] updating image-data
 D  [c2.qti.avc.decoder#870:2D-BB-Output] updating stride = 1536, width: 1280, height: 1024
 D  [c2.qti.avc.decoder#870:2D-BB-Output] updating vstride = 1024
 V  input buffer 0
 V  format changed to: android._color-format: int32(2135033992), android._video-scaling: int32(1), android._dataspace: int32(257), color-standard: int32(2), color-range: int32(1), color-transfer: int32(3), sar-height: int32(1), rotation-degrees: int32(90), hdr-static-info: data, sar-width: int32(1), crop: Rect(0, 0, 1279, 1023), width: int32(1280), feature-secure-playback: int32(0), frame-rate: int32(30), height: int32(1024), max-height: int32(4320), max-width: int32(8192), mime: string(video/raw), priority: int32(0), vendor.qti-ext-dec-timestamp-reorder.value: int32(1), color-format: int32(21), image-data: data, stride: int32(1536), slice-height: int32(1024)}
 V  input buffer -1
 D  [c2.qti.avc.decoder#870:Output[N]] updating image-data
 D  [c2.qti.avc.decoder#870:Output[N]] updating stride = 1280, width: 1280, height: 1024
 D  [c2.qti.avc.decoder#870:Output[N]] updating vstride = 1024
 V  colorFormat: 21
 V  buffer size: 2359040
 V  width (frame): 1280
 V  stride (frame): 1536
 V  height (frame): 1024

when

 D  [c2.qti.avc.decoder#870:2D-BB-Output] updating image-data
 D  [c2.qti.avc.decoder#870:2D-BB-Output] updating stride = 1536, width: 1280, height: 1024
 D  [c2.qti.avc.decoder#870:2D-BB-Output] updating vstride = 1024

called, can see "format changed" log too

format changed to: android._color-format: int32(2135033992), android._video-scaling: int32(1), android._dataspace: int32(257), color-standard: int32(2), color-range: int32(1), color-transfer: int32(3), sar-height: int32(1), rotation-degrees: int32(90), hdr-static-info: data, sar-width: int32(1), crop: Rect(0, 0, 1279, 1023), width: int32(1280), feature-secure-playback: int32(0), frame-rate: int32(30), height: int32(1024), max-height: int32(4320), max-width: int32(8192), mime: string(video/raw), priority: int32(0), vendor.qti-ext-dec-timestamp-reorder.value: int32(1), color-format: int32(21), image-data: data, stride: int32(1536), slice-height: int32(1024)}

However, it seems that even though the stride value changes when "updating image-data" is called again, this updated value is not being properly reflected in the stride. so i can see this log

 V  width (frame): 1280
 V  stride (frame): 1536
 V  height (frame): 1024

even stride is updated 1280

• error Logs

pid: 15900, tid: 16527, name: pool-47-thread-  >>> myApplicationName <<<
#00 pc 00000000008bc0e8  /data/app/~~NT0_FOgwIBLAp7mXKia1Iw==/<myApplicationName>-EBcFfWoWFVVeufPvkpyA6g==/lib/arm64/libopencv_java4.so (BuildId: 8c49264d6e0302480dbef69b206d784d4aa016e2)
#01 pc 000000000079c068  /data/app/~~NT0_FOgwIBLAp7mXKia1Iw==/<myApplicationName>-EBcFfWoWFVVeufPvkpyA6g==/lib/arm64/libopencv_java4.so (BuildId: 8c49264d6e0302480dbef69b206d784d4aa016e2)
#02 pc 000000000079be88  /data/app/~~NT0_FOgwIBLAp7mXKia1Iw==/<myApplicationName>-EBcFfWoWFVVeufPvkpyA6g==/lib/arm64/libopencv_java4.so (BuildId: 8c49264d6e0302480dbef69b206d784d4aa016e2)
#03 pc 00000000007a99e4  /data/app/~~NT0_FOgwIBLAp7mXKia1Iw==/<myApplicationName>-EBcFfWoWFVVeufPvkpyA6g==/lib/arm64/libopencv_java4.so (BuildId: 8c49264d6e0302480dbef69b206d784d4aa016e2)
#04 pc 00000000007a954c  /data/app/~~NT0_FOgwIBLAp7mXKia1Iw==/<myApplicationName>-EBcFfWoWFVVeufPvkpyA6g==/lib/arm64/libopencv_java4.so (BuildId: 8c49264d6e0302480dbef69b206d784d4aa016e2)
#05 pc 0000000001240874  /data/app/~~NT0_FOgwIBLAp7mXKia1Iw==/<myApplicationName>-EBcFfWoWFVVeufPvkpyA6g==/lib/arm64/libopencv_java4.so (tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop(tbb::internal::context_guard_helper<false>&, tbb::task*, long)+424) (BuildId: 8c49264d6e0302480dbef69b206d784d4aa016e2)
#06 pc 000000000123fefc  /data/app/~~NT0_FOgwIBLAp7mXKia1Iw==/<myApplicationName>-EBcFfWoWFVVeufPvkpyA6g==/lib/arm64/libopencv_java4.so (tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all(tbb::task&, tbb::task*)+352) (BuildId: 8c49264d6e0302480dbef69b206d784d4aa016e2)
#07 pc 000000000123d5c0  /data/app/~~NT0_FOgwIBLAp7mXKia1Iw==/<myApplicationName>-EBcFfWoWFVVeufPvkpyA6g==/lib/arm64/libopencv_java4.so (tbb::internal::generic_scheduler::local_spawn_root_and_wait(tbb::task*, tbb::task*&)+228) (BuildId: 8c49264d6e0302480dbef69b206d784d4aa016e2)
#08 pc 00000000007a94c8  /data/app/~~NT0_FOgwIBLAp7mXKia1Iw==/<myApplicationName>-EBcFfWoWFVVeufPvkpyA6g==/lib/arm64/libopencv_java4.so (BuildId: 8c49264d6e0302480dbef69b206d784d4aa016e2)
#09 pc 00000000007a93d0  /data/app/~~NT0_FOgwIBLAp7mXKia1Iw==/<myApplicationName>-EBcFfWoWFVVeufPvkpyA6g==/lib/arm64/libopencv_java4.so (BuildId: 8c49264d6e0302480dbef69b206d784d4aa016e2)
#10 pc 00000000012348d4  /data/app/~~NT0_FOgwIBLAp7mXKia1Iw==/<myApplicationName>-EBcFfWoWFVVeufPvkpyA6g==/lib/arm64/libopencv_java4.so (tbb::interface7::internal::task_arena_base::internal_execute(tbb::interface7::internal::delegate_base&) const+584) (BuildId: 8c49264d6e0302480dbef69b206d784d4aa016e2)
#11 pc 00000000007a9170  /data/app/~~NT0_FOgwIBLAp7mXKia1Iw==/<myApplicationName>-EBcFfWoWFVVeufPvkpyA6g==/lib/arm64/libopencv_java4.so (BuildId: 8c49264d6e0302480dbef69b206d784d4aa016e2)
#12 pc 000000000079b89c  /data/app/~~NT0_FOgwIBLAp7mXKia1Iw==/<myApplicationName>-EBcFfWoWFVVeufPvkpyA6g==/lib/arm64/libopencv_java4.so (cv::parallel_for_(cv::Range const&, cv::ParallelLoopBody const&, double)+620) (BuildId: 8c49264d6e0302480dbef69b206d784d4aa016e2)
#13 pc 00000000008b45c4  /data/app/~~NT0_FOgwIBLAp7mXKia1Iw==/<myApplicationName>-EBcFfWoWFVVeufPvkpyA6g==/lib/arm64/libopencv_java4.so (BuildId: 8c49264d6e0302480dbef69b206d784d4aa016e2)
#14 pc 00000000008b43e4  /data/app/~~NT0_FOgwIBLAp7mXKia1Iw==/<myApplicationName>-EBcFfWoWFVVeufPvkpyA6g==/lib/arm64/libopencv_java4.so (BuildId: 8c49264d6e0302480dbef69b206d784d4aa016e2)
#15 pc 00000000008b7d90  /data/app/~~NT0_FOgwIBLAp7mXKia1Iw==/<myApplicationName>-EBcFfWoWFVVeufPvkpyA6g==/lib/arm64/libopencv_java4.so (cv::hal::cvtTwoPlaneYUVtoBGR(unsigned char const*, unsigned long, unsigned char*, unsigned long, int, int, int, bool, int)+164) (BuildId: 8c49264d6e0302480dbef69b206d784d4aa016e2)
#16 pc 00000000008b9d14  /data/app/~~NT0_FOgwIBLAp7mXKia1Iw==/<myApplicationName>-EBcFfWoWFVVeufPvkpyA6g==/lib/arm64/libopencv_java4.so (BuildId: 8c49264d6e0302480dbef69b206d784d4aa016e2)
#17 pc 00000000008976e0  /data/app/~~NT0_FOgwIBLAp7mXKia1Iw==/<myApplicationName>-EBcFfWoWFVVeufPvkpyA6g==/lib/arm64/libopencv_java4.so (cv::cvtColor(cv::_InputArray const&, cv::_OutputArray const&, int, int)+688) (BuildId: 8c49264d6e0302480dbef69b206d784d4aa016e2)
#18 pc 0000000000e495c8  /data/app/~~NT0_FOgwIBLAp7mXKia1Iw==/<myApplicationName>-EBcFfWoWFVVeufPvkpyA6g==/lib/arm64/libopencv_java4.so (BuildId: 8c49264d6e0302480dbef69b206d784d4aa016e2)
#19 pc 0000000000e2e298  /data/app/~~NT0_FOgwIBLAp7mXKia1Iw==/<myApplicationName>-EBcFfWoWFVVeufPvkpyA6g==/lib/arm64/libopencv_java4.so (cv::VideoCapture::retrieve(cv::_OutputArray const&, int)+84) (BuildId: 8c49264d6e0302480dbef69b206d784d4aa016e2)
#20 pc 0000000000e2e3bc  /data/app/~~NT0_FOgwIBLAp7mXKia1Iw==/<myApplicationName>-EBcFfWoWFVVeufPvkpyA6g==/lib/arm64/libopencv_java4.so (cv::VideoCapture::read(cv::_OutputArray const&)+96) (BuildId: 8c49264d6e0302480dbef69b206d784d4aa016e2)
#21 pc 00000000006073c8  /data/app/~~NT0_FOgwIBLAp7mXKia1Iw==/<myApplicationName>-EBcFfWoWFVVeufPvkpyA6g==/lib/arm64/libopencv_java4.so (Java_org_opencv_videoio_VideoCapture_read_10+64) (BuildId: 8c49264d6e0302480dbef69b206d784d4aa016e2)
#27 pc 0000000000256f08  /data/app/~~NT0_FOgwIBLAp7mXKia1Iw==/<myApplicationName>-EBcFfWoWFVVeufPvkpyA6g==/base.apk (org.opencv.videoio.VideoCapture.read+0)
#32 pc 000000000002aaf4  /data/data/<myApplicationName>/code_cache/.overlay/base.apk/classes13.dex (<myApplicationName>.modules.video.VideoRetriever.retrieveOriginalsAndSaveForAnalyze$lambda$0+0)
#37 pc 000000000002aad8  /data/data/<myApplicationName>/code_cache/.overlay/base.apk/classes13.dex (<myApplicationName>.modules.video.VideoRetriever.$r8$lambda$iyzCBqgh6L_JopvZ7sX2bvdcvw4+0)
#42 pc 000000000002a8c4  /data/data/<myApplicationName>/code_cache/.overlay/base.apk/classes13.dex (<myApplicationName>.modules.video.VideoRetriever$$ExternalSyntheticLambda0.call+0)

It has been confirmed that this issue also occurs in the latest version, 4.10.0.

Steps to reproduce

        ...
        val videoCapture = VideoCapture(path)
        val mat = Mat()

        try {
            ...
            while (videoCapture.isOpened) {

                val readTask = Callable {
                    videoCapture.read(mat)
                    // use videoCapture.read(Matrix)
                }

                frameExtractorFuture = executor?.submit(readTask)
                ...
                try {
                    readSuccessful = frameExtractorFuture?.get(FRAME_EXTRACT_TIME_OUT_MILLIS, TimeUnit.MILLISECONDS) ?: false
                } catch (e: TimeoutException) {
                    logD(TAG_DEBUG, "frameExtract timeout exception : $e")
                    frameExtractorFuture?.cancel(true)
                    throw e
                } catch (e: Exception) {
                    throw e
                }
                ...
            }
        } catch (e: Exception) {
            throw e
        } finally {
            ...
        }

It seems that the error occurs at the "videoCapture.read(mat)"

Issue submission checklist

• I report the issue, it's not a question
• I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
• I updated to the latest OpenCV version and the issue is still there
• There is reproducer code and related data files (videos, images, onnx, etc)

[asmorkalov]

@Dev-Czle How do you get the video? Could you reproduce the issue with video-recorder sample? The same makes recording with VideoWriter and then plays back it with VideoCapture. Could you attach small video to reproduce the issue?

[Dev-Czle]

@asmorkalov
Sorry for the delay.
Here is the link to the video where the issue is reproduced. Download(https://drive.google.com/file/d/1-I3FaE7ckf8f_hqwov15GCVrL-MAgqgR/view?usp=drive_link)
The video has been compressed as its information may change.

The video's information is as follows:

Width: 1024
Height: 1280
Rotate: 90
Video codec: H.264
Size: 722KB
Device : Galaxy ZFlip3
OS : 14
OpenCV : 4.9.0
When using OpenCV 4.9.0 on a Galaxy ZFlip3, the app crashes upon calling VideoCapture.read(Matrix) with this video. The logs are consistent with the previously mentioned issue.

Additionally, I conducted tests with various versions.

Previous version: 4.5.5

Target version: 4.9.0

Tested versions:

• 4.5.5 -> Success
• 4.6.0 -> Success
• 4.7.0 -> Crash
• 4.8.1 -> Crash
• 4.9.0 -> Crash
• 4.10.0 -> Crash

The error logs consistently showed that the stride was longer than the width. I hope this information is helpful to you.

[asmorkalov]

Relates to changes in #22227

[asmorkalov]

@Dev-Czle I found couple of suspicious places and tries to reproduce the issue locally. Could you attach complete logcal log for the crash case?

[Dev-Czle]

@asmorkalov
Sorry for the delay.
Here is the complete crash log.(https://drive.google.com/file/d/1QraEVop_NskL2ww_AQa1qg0x0xCP5d6U/view?usp=sharing) The steps are as follows:

1. Copy a video from the Android gallery to a temporary folder
2. Perform transcoding using Android Media3
3. Retrieve the necessary metadata from the media using MediaExtractor
4. Start OpenCV operations with the video
5. Crash occurs at the OpenCV.read(Matrix) part
   There are also error logs related to threads, so I am sending those as well.

Even though I synchronized and used a single thread suspecting it might be a thread collision issue, the errors still occur.

[Dev-Czle]

@asmorkalov excuse me, did u check crash logs?

[asmorkalov]

Not yet, sorry. Will try to handle it in mean time.

[alexlyulkov]

@Dev-Czle I've created a small app, that plays your video, and it works fine. Could you please test it on your device? The project should work in Android Studio without any changes.

VideoReader.zip

[asmorkalov]

Just tried with Xiaomi mi 10 and Samsung Galaxy S10. The attached code works fine with both devices.

[Dev-Czle]

@alexlyulkov thanks. i will check it by my devices and tell you soon :)

Not all devices experienced crashes with my videos. The issue occurred on specific devices, such as the Galaxy ZFlip3 or Google Pixel 7. I also confirmed that the issue happened on the Galaxy S24. However, the issue occurred with videos other than the one I sent you. I am quite disappointed about this as well.