High resolution image capture problem

[sbkirby]

Hey @jeffbass,
I've been experimenting with high resolution image capture, and noticed a problem that is prevalent during this mode. It appears as though imagenode is writing over the images, and sending the same image two or three times in a row. I added some code to print a timestamp on each image for troubleshooting, and expected to get an image with the same timestamp on each of the duplicate images. Well, I'm getting the same timestamp(s), but it appears as though several timestamps are superimposed on top of one another. Not what I expected. This same code works as expected on RPi Zero's with a resolution of (640,480) @ 30 fps. I've attached a couple of example of images.
I'm using a YAML like the bird_cam_rpi.yaml with a RPi 4B, 2GB memory and RPi HQ 12M pixel camera.

Thanks
Stephen

[jeffbass]

Hi Stephen,
It definitely looks like duplicate frames to me, which is a bug in the imagenode code. I don't have a good answer for this one. I have a few theories, but I am not sure how to test them, since I don't have a test setup like yours. I'll describe my theories to give you some things to think about / investigate.

1. One possible issue is the way that imutils.video.Videostream() does threaded updates. It's not a bug, but by design, the PiVideoStream.update method updates self.frame continuously and the PiVideoStream.read method grabs whatever the current value of self.frame is at the time it is called. It is possible to get the same frame twice depending on settings of framerate and how fast PiVideoStream.read is called. I have had this happen when the framerate was quite different from the read rate. I have never used the RPi HQ cam and don't know if that is an issue. Adjusting the framerate to a lower value that is similar to the framerate of the entire pipeline might help.

2. A more likely issue is the detect_motion method in the Detector class in imaging.py. I designed it mainly for my water meter; it appends frames to send_q when state changes to motion and appends frames when state changes to still. If they happen close together, duplicate images can be appended. Given your application, it would make more sense to append frames only when motion is detected. Also, the code only has logic to set one send_count which is used to send that many images when a state change occurs. There should probably be a separate send_count for motion and for still, so that the send_count for still could be set to 0. Also, there is no checking for duplicate (overlapping) images being added when the motion and still events happen close together; that's a bug, but it doesn't show up in my application, so I haven't seen it happen or tried to fix it. But my hunch is that it is causing your duplicate images.

3. The detector settings for send_count, min_motion_frames and min_still_frames will affect the overlap. You may want to try a send_count: 2, a min_motion_frames: 2, min_still_frames=5 and see if you have fewer overlapping (duplicated) frames.

I think you have discovered a subtle bug in the detect_motion method. It is probably not the PiVideoStream frame rate, but I mentioned it because it has affected my own testing in the past. Threaded updates are always a bit tricky.

Changing the detector settings is something to experiment with. Start with the settings I suggested and move them up from there. I'd try that first.

I think the easiest way to try to approach (2), the possible subtle bug in the detect_motion() method, is to rewrite its append logic so that it appends frames only when state changes from still state to motion state (and NOT from motion state to still state). That is, only append frames when motion is detected (and not when "still" is detected). My water meter project needed the still frames; but it is probably the cause of your duplicate frames. In fact, re-reading the detect_motion code after not having looked at it in quite a while, I think it should have been called "detect_state_change" instead of "detect_motion". It would be simpler and more reliable to send frames when motion is detected, and NOT send any frames when "still" is detected.

It was a good idea to put timestamps on your images; the timestamps in the imagehub log and in the imagehub image file names are the times of receipt at the imagehub (which are NOT the times the images are taken).

I apologize, but I don't have the bandwidth to debug / re-write the detect_motion method right now. I'm not sure I can easily duplicate your problem with my setup. I hope you can try to fix it. If you do, it might be easiest to create a new "improved_detect_motion" method, so some A vs. B testing of the 2 methods could be done. I won't likely be able to revisit the detect_motion method for a while.

Thanks for the detailed bug report & good luck with any bug chasing. I'd appreciate any updates on what you find out and/or fix.
Jeff

[sbkirby]

Hey Jeff,
Sorry...This a puzzling bug. Thanks for your guidance. This gives me some spots to focus and troubleshoot. I've posted the code I'm using to timestamp the images for your review. It can be found at:

https://github.com/sbkirby/imagenode/blob/master/imagenode/tools/imaging.py

Adding the following to the detector/motion section of your YAML file will activate the timestamp.

        draw_time: ((0,255,255),2)
        draw_time_org: (5,5)
        draw_time_fontScale: 1 

[sbkirby]

I'm thinking out loud: If the following condition has been satisfied, STOP testing and just send send_count frames and then resume testing again. Note: The same would be true for 'still' conditions as well.

if self.moving_frames >= self.min_motion_frames:

I'll try some code to implement this idea.

[jeffbass]

Thanks...I took a look at your draw_time code. It looks really good. That would be a great pull request when you have time!

And your idea on just sending send_count frames before resuming testing sound like a good one!

[sbkirby]

Apparently there may be a problem, as you mentioned, with imutils VideoStream and repeating frames.

Possible getting repeated frames from VideoStream #213 - PyImageSearch/imutils#213

[sbkirby]

I set the send_count: 2, a min_motion_frames: 2, min_still_frames=5, and didn't see any difference in the overlapped timestamp.

[jeffbass]

The imutils VideoStream repeating frames behavior also gets mentioned in this blog post: ; Adrian explains it if you scroll down to comment of Jan 25, 2018 or search word "repeated" on that web page. I do not think that imutils VideoSteam is the cause of your duplicate frames, though; I just mentioned it because it may become relevant debugging debugging and testing. I ran into it 2 years ago when I was doing testing of the the first draft of the detect_motion method and was stepping through code lines and frames at a very slow rate.

Thanks for testing the settings changes and ruling that out as a possible cause.

[sbkirby]

Thanks for the info. I appreciate you researching this problem.
Stephen