Image whitening #96

jlecoeur · 2017-08-31T12:58:41Z

This is a follow-up of #90. I rebased @lericson's work to latest master.

On top of that I added a third whitening mode for param IMAGE_WHITEN:

IMAGE_WHITENING_DISABLED (=0) : equivalent to master
IMAGE_WHITENING_ALWAYS (=1) : performs image whitening before computing optic flow
IMAGE_WHITENING_AUTO (=2) : performs whitening before computing optic flow, if flow quality is lower than param IMAGE_WHIT_QTHRES, then recompute optic flow on the non-whitened image, use the optic flow with highest quality.

I also added a debug message with the average DT. It gives an indication of the computation time added by each method:

IMAGE_WHITENING_DISABLED: DT between 6.4ms and 6.6 ms
IMAGE_WHITENING_ALWAYS: DT between 8.0ms and 8.2ms
IMAGE_WHITENING_AUTO: DT between 8.0ms and 8.2ms with the default IMAGE_WHIT_QTHRES=150, i.e. it was not never recomputing optic flow. By artificially raising IMAGE_WHIT_QTHRES to 256, I forced it to always recompute optic flow, and then DT was between 14.2ms and 14.4ms. But keep in mind that in real scenario it would recompute optic flow only for few frames (only when the flow quality is bad). I guess that in these cases it is better to have measurements at lower update rate than no measurement.

FYI people from previous conversation @lericson @jgoppert @ChristophTobler @mhkabir @LorenzMeier

I have the same conclusions as in the previous PR: On the low textured floor that you can see in the pictures of #90, I have no optic flow with IMAGE_WHITENING_DISABLED. The quality is very good with IMAGE_WHITENING_ALWAYS and IMAGE_WHITENING_AUTO. I could not test IMAGE_WHITENING_AUTO because the weather is bad today and it only kicks in when the luminosity varies a lot in the image.

@ChristophTobler @lericson can you try with all three modes and report how it works for you?

We noticed quite heavy flickering in the image output from the camera sensor. We tested an array of approaches to mitigate the issue. Primarily we perform whitening of the camera input. This improves the flow quality and stability immensely for us.

- Added parameter PARAM_IMAGE_WHITENING: * 0: no whitening * 1: always use whitening * 2: use whitening, if flow quality is low, recompute flow with non-whitened images - Added parameter PARAM_IMAGE_WHITENING_QUALITY_THRESHOLD (default 150): value of optic flow quality bellow which optical flow is recomputed from non-whitening images - Added separate buffers for whitened images

jlecoeur · 2017-08-31T13:00:30Z

Note: In mode IMAGE_WHITENING_AUTO, I chose to use whitened image first. But we can discuss whether non-whitened image should be tried first, and whitening performed only if optic flow quality is too low. It would be advantageous because that would not add the computation overhead of whitening when flying over well-textured floor. However when indoors on low textured floor, it would recompute optic flow all the time!

ChristophTobler · 2017-09-08T13:22:21Z

Thanks! I'll have a look at it as soon as possible.

ChristophTobler · 2017-09-15T08:57:04Z

@jlecoeur Good work! I just had go with this PR and did some bench testing. The weird artifacts from #90 (comment) are now gone. Generally it seems to work well with my office conditions.
What I don't like is that now the quality is pretty much always at the max even when the flow just outputs noise. This is dangerous because that way the estimator will always trust the computed flow even though it shouldn't. Can you think of a method to prevent that?

jlecoeur · 2017-09-15T13:33:11Z

@ChristophTobler Indeed. My understanding is that when the raw image is almost flat/featureless, image whitening is increasing image noise, so all blocks have high gradients after whitening, even though this is mostly noise. Thus all blocks pass the test with gradient larger than PARAM_BOTTOM_FLOW_FEATURE_THRESHOLD here. Maybe we can add another check to also discard blocks that have a too large gradient?

Another issue is that because whitened images generally have larger gradients, the thresholds PARAM_BOTTOM_FLOW_VALUE_THRESHOLD and PARAM_BOTTOM_FLOW_FEATURE_THRESHOLD need to be different in the whitened and non-whitened cases.

jlecoeur · 2017-09-15T13:43:28Z

@ChristophTobler Another idea: when images are mostly noise, the NUM_BLOCKS*NUM_BLOCKS optic flow estimates must have large standard deviation.
Maybe we can adjust flow quality using the std dev of flow measurements?

LorenzMeier · 2017-09-16T11:36:16Z

Sounds like a good starting point. Would you have time next week to have a go at it?

jlecoeur · 2017-09-16T17:09:31Z

@LorenzMeier That worked!
It now draws optic flow vectors on the image. Apart from being fancy, it shows that flow has large std dev when the image is grey. I use std dev to scale the quality with qual *= (SEARCH_SIZE - stddev) / SEARCH_SIZE).

ChristophTobler · 2017-09-18T07:39:42Z

@jlecoeur great work! Generally works very good for me! I have done a small change (see here) that restricts flow a bit more. Would be good if you could have a look at that.

jlecoeur · 2017-09-18T10:06:45Z

I like that quality is now a more continuous indicator. However we have to be careful that optic flow is not rejected too often by filters on the PX4 side.
@ChristophTobler I think LPE_FLW_QMIN should be lowered (it is 150 by default).
With these changes the quality is not easily above 150, and it looks still usable at quality=40. Ideally quality should be taken into account by the filters, right?

…lity

ChristophTobler · 2017-09-18T10:50:32Z

@jlecoeur I agree. I would adjust the parameter in the Firmware. Probably between 20-50 depending on the setup/environment.

ChristophTobler · 2017-09-18T10:52:39Z

@jlecoeur It actually already is very low https://github.com/PX4/Firmware/blame/master/src/modules/ekf2/ekf2_params.c#L684

jlecoeur · 2017-09-18T11:01:20Z

Yes EKF2 will be fine but not LPE
https://github.com/PX4/Firmware/blame/master/src/modules/local_position_estimator/params.c#L57

ChristophTobler · 2017-09-18T11:03:07Z

Yeah we should lower that, once this PR is in. I still want to do some actual flight tests.

ChristophTobler · 2017-09-18T14:17:12Z

@jlecoeur Here's a log of a test flight https://logs.px4.io/plot_app?log=458daba2-d4d1-4e01-8ac9-6da289620e28. Worked nicely! Maybe I can get another test when it's a little darker outside.

jlecoeur · 2017-09-18T14:54:27Z

Nice thanks!
Just noting: flow quality was 250 most of the flight, never 255, and it dropped to 150~200 when you did the lateral steps + two points at quality=0. Looks good.

ChristophTobler · 2017-09-20T13:02:39Z

@jlecoeur

Just noting: flow quality was 250 most of the flight, never 255, and it dropped to 150~200 when you did the lateral steps + two points at quality=0. Looks good.

That's fine.

Did you do some test flights in a environment with mixed lighting conditions?

AlexisTM · 2017-10-06T10:20:56Z

This is awesome.

With small changes (25 feature track, always whitening, all filtering on) I have a (slowly drifting) position. As I recalibrate with the higher level control software, I do not care about having any drift, but I care about having a position to be able to make safe autonomous takeoff.

It was impossible with the master while perfectly working here. The drift (when not moving) is about 10cm/s while when I had no position and I regain it, it was sometimes some crazy 10m/s and we had multiple crashes.

global_data.param[PARAM_SYSTEM_SEND_LPOS] = 0; //1
global_data.param[PARAM_IMAGE_WHITENING] = IMAGE_WHITENING_ALWAYS; // IMAGE_WHITENING_AUTO
global_data.param[PARAM_SONAR_FILTERED] = 1; // 0
global_data.param[PARAM_BOTTOM_FLOW_FEATURE_THRESHOLD] = 25; // 50
global_data.param[PARAM_BOTTOM_FLOW_HIST_FILTER] = 1; // 0
global_data.param[PARAM_BOTTOM_FLOW_GYRO_COMPENSATION] = 1; // 0
global_data.param[PARAM_BOTTOM_FLOW_LP_FILTERED] = 1; // 0
global_data.param[PARAM_EXPOSURE_MAX] = 125; // 500
global_data.param[PARAM_GAIN_MAX] = 8; // 16
global_data.param[DEBUG_VARIABLE] = 0; // 1

@jlecoeur I am not sure about the IMAGE_WHITENING_AUTO mode logic. I would prefer it to be a state machine which switches from "prefered white or normal".

The actual implementation always computes the whitened image and if the whitened is too bad, try the other one.

quality_whitened = compute_on_whitened_img()

if quality < threshold:
    quality_normal = compute_on_normal_img()
    if quality > quality_whitened: 
        used_whitened_image = true
    else:
       used_whitened_image = false
else:
   used_whitened_image = true

I would propose to use use_whitened_image as a state variable to switch from always white or always normal.

quality_white = 0
quality_normal = 0

if use_whitened_image:
    quality_white = compute_on_whitened_img()
    if quality_white < threshold:
    	quality_normal = compute_on_normal_img()
else:
    quality_normal = compute_on_normal_img()
    if quality_normal < threshold:
    	quality_white = compute_on_whitened_img()

use_whitened_image = (quality_white >= quality_normal)

jlecoeur · 2017-10-06T10:35:45Z

@AlexisTM good to hear that you had good results with it! If you could post a log that would be awesome.

With small changes (25 feature track, always whitening, all filtering on)

Do you think some of these changes should be the default?

I am not sure about the IMAGE_WHITENING_AUTO mode logic

In practice quality_whitened is always higher than quality_normal except in corner cases like mixed bright/dark images. So the idea is to check first the whitened images to make sure that the highest quality is selected.
With what you propose, there would be less mode switching. However it is possible that normal images are selected (use_whitened_image stays = 0) when quality_whitened > quality_normal > threshold, which is not optimum.

AlexisTM · 2017-10-06T10:41:54Z

Do you think some of these changes should be the default?

@jlecoeur I don't think it would be the default except for DEBUG_VARIABLE = 0; Indeed, I did not do a lot of research/test to get it working, I changed it to what I think would be better for me in this application.

In practice quality_whitened is always higher than quality_normal

Are you sure it is the case on a sunny day outside? Indoor, it is indeed the case.

ChristophTobler · 2017-10-06T11:36:57Z

@AlexisTM

It was impossible with the master while perfectly working here. The drift (when not moving) is about 10cm/s while when I had no position and I regain it, it was sometimes some crazy 10m/s and we had multiple crashes.

10cm/s is already quite bad! Are you doing hand held tests when you say "not moving"? What is the environment like that you test in? Some logs would be good.

AlexisTM · 2017-10-06T12:06:53Z

@ChristophTobler I personally think that 10cm/s is pretty good as before I could only get a position at least 30/40cm from the ground. The sensor is at about 10cm of the ground, so the distance sensor is not at a valid distance (< 30 cm).

You can still notice the bricks used before to fly from higher. (We do with what we have)

On the ground: https://review.px4.io/plot_app?log=b902dd0e-70f6-4c06-bea1-2de5c0d25be8

Flight: Takeoff, hover, land - https://review.px4.io/plot_app?log=e3d447b1-bcde-455c-bba2-a8588b35a481

ChristophTobler · 2017-10-06T12:28:31Z

@AlexisTM Thanks!

These seem a bit high and there is an offset https://review.px4.io/plot_app?log=b902dd0e-70f6-4c06-bea1-2de5c0d25be8#Nav-Raw-Angular-Speed-Gyroscope. Have you checked the calibration and dampening?

I see you are flying offboard. But why are there no setpoints https://review.px4.io/plot_app?log=e3d447b1-bcde-455c-bba2-a8588b35a481#Nav-Local-Position-X?

Can you maybe also provide a video of a flight?

AlexisTM · 2017-10-06T12:47:21Z

@ChristophTobler

Video of the first flight - https://photos.app.goo.gl/AcevYPa7PVNV6tRF2

PS: I could not get the log of this flight due to a bug in QGC when too many logs are available or due to different log versions (switched to APM, PX4, different versions) and I did not take a video of the second flight.

Also, the offboard control is done via velocity setpoints.

mhkabir · 2018-04-28T16:19:59Z

@AlexisTM @jlecoeur @ChristophTobler Any updates on this one? Did we ever get this branch working correctly? Could someone summarise?

jlecoeur · 2018-04-30T07:56:34Z

I used this branch with success. It improves flow quality over surfaces with poor contrast. The reported flow quality is more linear with the actual flow quality, so LPE default setting should be updated to reduce the flow quality threshold.

AlexisTM · 2018-05-08T12:32:14Z

@mhkabir I am using my master: https://github.com/umdlife/Flow which works fine on a low light condition but has a change that should not be on the master.

It is this PR + few changes:

Compatibility with another toolchain version (7.2.1)
Enable all filters
Reduce the quality threshold

This last change is not viable in production. It returns a quality of almost always 255 and therefore drifts more but we correct it using tags. Right now we are working on 1.6.5 which has a bug when the PX4Flow is not valid. (We were facing a bug on the v1.7.3 (Pixhawk related)).

Full changes:

Added 7.2.1 GNU toolchain: makefiles/baremetal/toolchain_gnu-arm-eabi.mk

+CROSSDEV_VER_SUPPORTED	 = 4.7.4 4.7.5 4.7.6 4.8.2 4.8.4 4.9.3 5.4.1 7.1.0 7.2.0 7.2.1

Difference in settings: src/modules/flow/settings.c:

global_data.param[PARAM_SYSTEM_SEND_LPOS] = 0; //1
global_data.param[PARAM_BOTTOM_FLOW_HIST_FILTER] = 1; // 0
global_data.param[PARAM_IMAGE_WHITENING] = IMAGE_WHITENING_ALWAYS; // IMAGE_WHITENING_AUTO
global_data.param[PARAM_SONAR_FILTERED] = 1; // 0
global_data.param[PARAM_BOTTOM_FLOW_FEATURE_THRESHOLD] = 25; // 50
global_data.param[PARAM_BOTTOM_FLOW_HIST_FILTER] = 1; // 0
global_data.param[PARAM_BOTTOM_FLOW_GYRO_COMPENSATION] = 1; // 0
global_data.param[PARAM_BOTTOM_FLOW_LP_FILTERED] = 1; // 0
global_data.param[PARAM_EXPOSURE_MAX] = 125; // 500
global_data.param[PARAM_GAIN_MAX] = 8; // 16
global_data.param[DEBUG_VARIABLE] = 0; // 1

victordias2000 · 2020-08-02T08:47:08Z

We're quite interested in the results that have been presented here as we are experiencing similar position drifting issues. Is there any timeline to get this PR merged? Is there any other way to attenuate the problems by not whitening the image?

@LorenzMeier

Thanks!

jlecoeur · 2020-08-02T09:53:14Z

Hi @victordias2000
I have not been using PX4Flow for a while, so I have not pushed this PR further. I have used it extensively with good results though. I think there is still value to merge. Let's rebase, flight test and merge.

Let me try to rebase in the next week.
@ChristophTobler do you still have a drone with PX4Flow ready for testing?

victordias2000 · 2020-08-03T07:43:52Z

That would be fantastic @jlecoeur !!!

Thanks.

AlexisTM · 2020-08-03T07:51:44Z

I personally do not have a PX4 drone here, thus cannot help on it. Yet, we used this firmware quite heavily to allow low light environment. The actual repo used was: https://github.com/umdlife/Flow

jlecoeur · 2023-09-19T07:11:11Z

This has been open for a long time without activity. I am closing this PR.

AlexisTM · 2023-09-19T08:22:35Z

It has been helpful to me at the time, thanks for the contribution @jlecoeur \o/

lericson and others added 11 commits August 31, 2017 10:53

Improve flow quality

0076278

We noticed quite heavy flickering in the image output from the camera sensor. We tested an array of approaches to mitigate the issue. Primarily we perform whitening of the camera input. This improves the flow quality and stability immensely for us.

Use floats for whitening

cfc4d6e

Add 4.8.2 as allowed compiler

e9d9e1f

Add 7.1.0 as allowed compiler

cc7fbe1

Add whitening parameter

326edfd

Optimize types of whitening variables

580844e

Misleading indentation

41c153d

Fix compile errors on hosts with python3 as default

dd2425b

Revert SONAR_TIMER_COUNT

e230142

Send average DT

0edbe5e

Allow GCC 7.2.0

f189d77

jlecoeur added 5 commits September 16, 2017 17:35

Use NUM_BLOCKS for size of arrays

06526c3

Avoid 2nd computation of compute_diff for optic flow drawing

692061a

Display flow points at center of blocks

79d1dd5

Add param USB_DRAW_FLOW to display optic flow vectors on image

ef15b31

Compute OF std dev to scale quality

9a136a6

jlecoeur and others added 2 commits September 18, 2017 11:42

USB: Send image with highest flow quality

cebbc4b

scale flow quality more restrictive

398e9c3

jlecoeur force-pushed the pr-image-whitening-auto branch from d75ce2a to 398e9c3 Compare September 18, 2017 09:43

Lower OF recomputation min quality threshold to adapt to stricter qua…

b4ddfb8

…lity

This was referenced Oct 9, 2017

High velocities due to the PX4Flow; making everything unstable. #97

Closed

BUG on EKF2 when the PX4Flow goes invalid PX4/PX4-Autopilot#8102

Closed

jlecoeur mentioned this pull request Apr 30, 2018

Unable to make #99

Closed

jlecoeur closed this Sep 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Image whitening #96

Image whitening #96

jlecoeur commented Aug 31, 2017

jlecoeur commented Aug 31, 2017

ChristophTobler commented Sep 8, 2017

ChristophTobler commented Sep 15, 2017

jlecoeur commented Sep 15, 2017

jlecoeur commented Sep 15, 2017

LorenzMeier commented Sep 16, 2017

jlecoeur commented Sep 16, 2017

ChristophTobler commented Sep 18, 2017

jlecoeur commented Sep 18, 2017 •

edited

Loading

ChristophTobler commented Sep 18, 2017

ChristophTobler commented Sep 18, 2017

jlecoeur commented Sep 18, 2017

ChristophTobler commented Sep 18, 2017 •

edited

Loading

ChristophTobler commented Sep 18, 2017

jlecoeur commented Sep 18, 2017

ChristophTobler commented Sep 20, 2017

AlexisTM commented Oct 6, 2017 •

edited

Loading

jlecoeur commented Oct 6, 2017

AlexisTM commented Oct 6, 2017 •

edited

Loading

ChristophTobler commented Oct 6, 2017

AlexisTM commented Oct 6, 2017 •

edited

Loading

ChristophTobler commented Oct 6, 2017

AlexisTM commented Oct 6, 2017 •

edited

Loading

mhkabir commented Apr 28, 2018

jlecoeur commented Apr 30, 2018

AlexisTM commented May 8, 2018

victordias2000 commented Aug 2, 2020 •

edited

Loading

jlecoeur commented Aug 2, 2020

victordias2000 commented Aug 3, 2020

AlexisTM commented Aug 3, 2020 •

edited

Loading

jlecoeur commented Sep 19, 2023

AlexisTM commented Sep 19, 2023

Image whitening #96

Image whitening #96

Conversation

jlecoeur commented Aug 31, 2017

jlecoeur commented Aug 31, 2017

ChristophTobler commented Sep 8, 2017

ChristophTobler commented Sep 15, 2017

jlecoeur commented Sep 15, 2017

jlecoeur commented Sep 15, 2017

LorenzMeier commented Sep 16, 2017

jlecoeur commented Sep 16, 2017

ChristophTobler commented Sep 18, 2017

jlecoeur commented Sep 18, 2017 • edited Loading

ChristophTobler commented Sep 18, 2017

ChristophTobler commented Sep 18, 2017

jlecoeur commented Sep 18, 2017

ChristophTobler commented Sep 18, 2017 • edited Loading

ChristophTobler commented Sep 18, 2017

jlecoeur commented Sep 18, 2017

ChristophTobler commented Sep 20, 2017

AlexisTM commented Oct 6, 2017 • edited Loading

jlecoeur commented Oct 6, 2017

AlexisTM commented Oct 6, 2017 • edited Loading

ChristophTobler commented Oct 6, 2017

AlexisTM commented Oct 6, 2017 • edited Loading

ChristophTobler commented Oct 6, 2017

AlexisTM commented Oct 6, 2017 • edited Loading

mhkabir commented Apr 28, 2018

jlecoeur commented Apr 30, 2018

AlexisTM commented May 8, 2018

victordias2000 commented Aug 2, 2020 • edited Loading

jlecoeur commented Aug 2, 2020

victordias2000 commented Aug 3, 2020

AlexisTM commented Aug 3, 2020 • edited Loading

jlecoeur commented Sep 19, 2023

AlexisTM commented Sep 19, 2023

jlecoeur commented Sep 18, 2017 •

edited

Loading

ChristophTobler commented Sep 18, 2017 •

edited

Loading

AlexisTM commented Oct 6, 2017 •

edited

Loading

AlexisTM commented Oct 6, 2017 •

edited

Loading

AlexisTM commented Oct 6, 2017 •

edited

Loading

AlexisTM commented Oct 6, 2017 •

edited

Loading

victordias2000 commented Aug 2, 2020 •

edited

Loading

AlexisTM commented Aug 3, 2020 •

edited

Loading