Add possibility to manually scale the process noise covariance matrix

[edblu1]

Hey!
We've been using the boxmot yolo_tracking repository with the OCSort tracker for a while, and we encountered a small issue. At lower framerates, like 5 fps, we noticed the Kalman Filter predictions weren't performing as well as we'd hoped. Specifically, the predictions seemed too strict and stiff, especially in dynamic environments where the bounding box sizes and positions change a lot.
After some investigation and experimentation, we found that adjusting the Q Matrix (process noise covariance) to increase the uncertainty/noise level significantly improved prediction performance in our tests. However, the current repository doesn't offer a straightforward way to make this adjustment, so we created this pull request to make it easier to adjust the process noise covariance matrix if needed.
As changes we added two new parameters to the OCSort class. One parameter scales the expected process noise of the center of the bounding box, and the other scales the process noise of the size of the bounding box. This way, users can fine-tune the Kalman Filter to better suit their specific scenarios. We've also added a pytest function to validate the proper functioning of our changes.
We've also included some example video snippets to showcase the suboptimal behavior and the improvement after the adjustment. In these clips, a car navigates a traffic circle, causing rapid changes in both direction and scale of the bounding box. The current Kalman filter setup struggles to handle these dynamic changes and thus seems to lag behind.
In the videos, the Kalman Filter prediction is shown in blue, while the detected bounding box is shown in yellow.

We'd be really happy if you could accept this pull request, but we totally understand if it's not something you want to incorporate into the main repository. In that case, we'll maintain our own fork. If you need any further adjustments in the code, please let us know, and we will incorporate them.

Thanks a bunch for considering this!

[mikel-brostrom]

At lower framerates, like 5 fps, we noticed the Kalman Filter predictions weren't performing as well as we'd hoped.

The parameters in this repo are optimized for MOT17 in which most sequences are 30FPS.

We'd be really happy if you could accept this pull request, but we totally understand if it's not something you want to incorporate into the main repository.

Seems like a reasonable PR to me 😄, for customization, specially at lower frame rates.

If you need any further adjustments in the code, please let us know, and we will incorporate them.

This request may be outside your research scope. But would you be interested in adding this to the BoTSORT KF? This would make the process noise searchable with the evolve.py script for all trackers given that:

• BoTSORT and ByteTrack share an almost identical xyah KF
• OCSORT, DeepOCSORT, HybridSORT share an almost identical xys KF

The separation and redundancy in the KFs has been due to lack of time. I can do the filter cleaning and adding the parameters to the genetic parameter search in a later PR 🚀

[mikel-brostrom]

Did you find these values being a good start for tracking at 5FPS in your experiments?

Q_xy_scaling = 0.01,
Q_s_scaling = 0.0001

[edblu1]

Did you find these values being a good start for tracking at 5FPS in your experiments?

Q_xy_scaling = 0.01,
Q_s_scaling = 0.0001

Ah no, this default initialization just ensures, that the KF behaves the same as before without the parameters.
In the original KF script, Q gets initialized as a identity matrix of size 7 and is then scaled by:

self.kf.Q[-1, -1] *= 0.01
self.kf.Q[4:, 4:] *= 0.01

resulting in the diagonal being: [1 1 1 1 0.01 0.01 0.0001]

So the behaviour of the KF only changes if the user actively adjusts Q_xy_scaling or Q_s_scaling.

[edblu1]

This request may be outside your research scope. But would you be interested in adding this to the BoTSORT KF? This would make the process noise searchable with the evolve.py script for all trackers given that:

• BoTSORT and ByteTrack share an almost identical xyah KF
• OCSORT, DeepOCSORT, HybridSORT share an almost identical xys KF

I haven't really looked into the KF in BoTSort because we're mainly using and researching OCSort right now. But your idea sounds great, and I'll check it out when I find the time! :)

The separation and redundancy in the KFs has been due to lack of time. I can do the filter cleaning and adding the parameters to the genetic parameter search in a later PR 🚀

That sounds great, thanks!! 🚀

[mikel-brostrom]

Ah no, this default initialization just ensures, that the KF behaves the same as before without the parameters.

Ohh. Yes, your are right, I just revisited the OCSORT KF.

I haven't really looked into the KF in BoTSort because we're mainly using and researching OCSort right now. But your idea sounds great, and I'll check it out when I find the time! :)

BoTSORT or ByteTrack KF, whichever you find easier to work with 😄

[mikel-brostrom]

I just refactored the KFs @edblu1! I still have to unify all KalmanBoxTrackers in OCSORT, DeepOCSORT and HybridSORT such that all of them can be made configurable! 🚀

[mikel-brostrom]

What where the values you got to, if I may ask:

Q_xy_scaling = 0.01,
Q_s_scaling = 0.0001

for your 5 FPS videos?

[edblu1]

I just refactored the KFs @edblu1! This could be ported to the new file called xysr_kf.py such that all the xysr-kf-based trackers (OCSORT, DeepOCSROT and HybridSORT) could benefit of this PR! 🚀

Great! 🚀
I will look into updating this PR with the change on the weekend!

What where the values you got to, if I may ask:

Q_xy_scaling = 0.01,
Q_s_scaling = 0.0001

for your 5 FPS videos?

We haven't tested that many configurations yet, but one that worked quite well was to keep Q as an identity Matrix. Of course there are scenarios where such a "high" uncertainty will not provide optimal tracking behaviour, especially in scenes with long occlusions.

[edblu1]

Hi! I have now finally updated the xysr Trackers OCSort and DeepOCSort to work with scalable Q Matrices.
I also updated the pytest to test the KalmanBoxTracker for both of them.

I noticed, that Hybridsort also incorporates a xysr KalmanFilter, but with 9 dimensions instead of 7.
I have not tested anything with Hybridsort yet, so I don't know how this would behave. Also it would have to be tested if it is a good idea to also let the user scale the new noise dimension (the uncertainty of the confidence of the tracklet ċ).

[mikel-brostrom]

I have not tested anything with Hybridsort yet, so I don't know how this would behave. Also it would have to be tested if it is a good idea to also let the user scale the new noise dimension (the uncertainty of the confidence of the tracklet ċ).

I agree. Let's keep this as it is for now. Good job on the PR! Merging

[mikel-brostrom]

I now added the possibility to search this parameter using the evolve script 🚀

[edblu1]

Great, Thanks for incorporating this so fast! 🚀