Discussion for further sensor filtering and algorithmic improvements

[agittins]

A continuation of the fruitful chat with @jaymunro in #99 which lead up to the v0.5.0 release today.

I've been finding the elimination of peaks to be pretty promising thus far.

[jaymunro]

Sorry, been caught up with various matters. But I do now have some metrics with the change in filtering.

As we see here, the outliers with Max (tesla) are greatly reduced after the Mar 18 update but area resolution is only mildly improved.

However the Suma (Microtik beacon) seems changed but still has quite a few outliers and area resolution is only mildly better if at all.

I think this needs a bit more of a detailed look which I hope to get to over the next few days.

[jaymunro]

As per the screenshot in #144 (also below), we see Zoe bouncing back and forth between the kitchen and dining room. Where she was sitting at the time was fairly equidistant between both proxies. However due to the nature of the proxies (they're Shellys in the light switches) they tend to be on one side of a room and not in the centre, so this may become quite a common occurrence. But location accuracy is a subject for another time.

To mitigate the bouncy times, I was thinking if we could determine the person is stationary and we are seeing this behaviour, then we could make the proxies a bit 'stickier', as in the area entity would require a higher difference in distance before switching back to the other.

I think this could be done by:

1. Recording the previous 3 proxies [A, B, C]
2. If the current proxy [B] or the previous proxy [A] is the same as the one just it is just about to switch to [A] or [C], then turn on a flag with those two or three proxies as attributes.
3. If the flag is on and an proxy switch is about to occur into an proxy not in the flag attributes, clear the flag
4. If the flag is on and an proxy switch is about to occur into an proxy in the flag attributes then incorporate the interval since the last switch into a stickiness attribute. The method of incorporating it may be an average or similar that would result in a value in metres that would increase with swaps and would be added to the distance of any proxy about to be switched to that is in the stickiness attributes.

The idea being that if swapping is frequent the stickiness goes up and as the frequency decreases, the stickiness average would decrease, kinda an inverse to the time * the average distance to the proxies * a fudge factor.

That way stationariness (gee, I didn't know that was a word) is determined by the frequency of swapping and the stickiness attribute auto regulates the swapping without interfering with moving to another proxy (area) nor interfering with actually moving closer to one of the proxies in the stickiness flag.

EDIT:
The stickiness value could simply be the flag. zero is clear, non-zero is the stickiness distance associated with the 2 or 3 proxies in the attributes.

[agittins]

Hmmm that's an interesting approach. It makes sense that entities that are "flapping" between areas could benefit from some damping to their ability to switch.

I've been thinking about this in the back of my mind while doing the filtering stuff, too. I'll give your idea some time to percolate. My current thinking is that a lot of this switching is (probably?) caused by the current proxy receiving occluded signals or missing packets, rather than the usurper proxy actually reading closer (I'll need some more data to confirm that) or at least, not reading much closer than it was previously.

I'm thinking it might work to compare the velocities implied by the two proxies (mathematically this might be exactly what you're suggesting, since we're integrating distance over time). If the challenging proxy implies a low "approaching" velocity, then we can assume it's probably stationary and the two proxies are reasonably equidistant or the incumbent proxy is giving poor quality readings right now. However if the usurper proxy is indicating a "high" "approach velocity" (as opposed to the "retreat velocity" that max_velocity enforces) then we can assume it really is getting closer to that area, and we should switch.

I think this might give more stability but without compromising on responsiveness, based on the plots I've seen while testing.

Does that sound like a decent approach to you? I'm leaning somewhat toward trying to model things more... Newtonianly(?!) as my previous area-deciding logic got pretty confusing to debug once I was several levels deep in comparing proxies using more "reasoning-based" approaches.

For what it's worth, I think the long-goal of trilateration will inherently assist with this, as it will start to "understand" the relative relation between proxies/areas, rather than treating them as a binary/ternary/(n-ary?) choice.

Could you gather some graphs of Suma's "distance to" measurements for the proxy's it tends to switch between, along with the "Area" sensor? So similar to the below, without the final "distance" sensor or any of the "unfiltered" sensors, and at a fine-enough timescale that I can see the per-interval updates during some of the decisions.

Here's a beacon I carried to and left on a table, roughly equidistant from kitchen and dining proxies. I left the area, and even with no motion nearby (there was someone a few metres the other side of the lounge proxy) we still get a fair amount of variability. (my rssi figures aren't well-calibrated at the moment, our dining/kitchen is rather significantly less than 100sqm!)

Interesting points:

• the "kitchen gap" is a long drink between packets. I can't see any evidence of the proxy going offline during that time, so it's a bit odd. It is running esphome 2023.9.1 so perhaps it's an older ble issue.
• A, B and C all resulted in (initial for A/C) area switches due to kitchen getting a a reading at a time that dining's reading was over 2 seconds old. There are cases where switching would be valid in that case, but I think the fact that Kitchen's "velocity" is zero for that and many prior readings should prevent the switch - even though dining's velocity is up, it's "retreating" which means less reliable, so Kitchen having an "approach velocity" less-than-or-equal-to nearly zero should prevent an area switch.
• A and C's second area switches are a little different, while their first switch seems to be based on "freshness" (ie stale readings from incumbents), but the second switches at A & C are "valid" (but not valid) case of kitchen now having a closer (or perhaps equal, for C?) range. But if we used the same velocity-based filter again, we would ignore the change because the usurping proxy still has a near-zero-or-negative "approach" velocity.
• D is tricky. It was caused by someone moving around in the adjacent room(!) We see the range on Kitchen still being very steady, while dining reads the beacon as retreating. I don't know what an ideal outcome should look like for D. The kitchen range is strong evidence for the beacon not moving (a stable signal is statistically less likely than a noisy signal, so it deserves extra trust), so should we try to conclude that the other readings are erroneous from noise (which is "relatively" true) or should we conclude the beacon was circling the dining beacon in an ideal (ie, impossible, but relevant) way? Once the kitchen reading is longer it's hard to make a case for "it hasn't moved" but the rock-solid range from dining really does imply that it hasn't - a device that really is moving should probably cause reasonable perturbation to all proxies readings due to reflections etc - unless it's super-close.

I think that similarly to how we have a max_(retreat)velocity for deciding if a retreating reading is valid for the incumbent proxy, a min_approach_velocity could decide if a proxy should win the area contest even if it is now the closest. I think it would have to be tempered with a timeframe limit of how long it can have the closest reading and a low velocity before it wins the contest.

At any rate, D is an issue, since it's hard to distinguish between "stationary beacon, gradual noise effects on proxies" versus "beacon circling a proxy" - I think I lean toward the former, for the most part.

Multiple proxies in each area really does solidify things though - with two proxies in my studio I basically never get spurious area switches for my watch, even though it is particularly prone to signal variations based on wrist position and orientation. Again, trilateration would also solve a certain amount of this, since it doesn't suffer from having to treat areas as being defined only by proxies "in" that area.

I'll be interested to see if my observations here line up with what you're seeing as well.

[jaymunro]

Could you gather some graphs of Suma's "distance to" measurements for the proxy's it tends to switch between, along with the "Area" sensor? So similar to the below, without the final "distance" sensor or any of the "unfiltered" sensors, and at a fine-enough timescale that I can see the per-interval updates during some of the decisions.

Suma is not switching much at all any more. Unless it is leaving there is only one proxy close by (the garage door opener). The next closest is the back garden esp32 (M5Stack) which is a bit temperamental. It seems to only want to communicate to one HA at a time, sometimes my production, sometimes my dev, whereas the Shellys will quite happily communicate to both.

I think this screen shows what you're looking for, but it also shows that sometimes the Shelly's reception will have unfiltered distances between 1.26m - 3.16m (left side of the graph) for many hours, and then when the other car leaves (4:42) it will swap to distances between 1.26 and 6.81 (right side of the graph), and when the other car returns it will change again (1.26-5.01m in the case of last night). I'm just mentioning this to give an example of how much things can change with a change in environment, but yet while everything is static (no one in garage and nothing moving) there are still huge variations in the signal.

Also note that the M5Stack esp32 is connected via WiFi but it is still monitoring Bluetooth. I believe the default settings in ESPHome (below) mean that every 320ms it will scan for bluetooth signals for only 30ms. This naturally results in many missed packets, hence the gaps in the 'back garden' graph. The Shellys don't seem to suffer from this, likely because they have multiple radios.
https://esphome.io/components/esp32_ble_tracker.html

esp32_ble_tracker:
  scan_parameters:
    interval: 320ms
    window: 30ms

Now that Private BLE is connected though, I can also add my watch or phone. In this graph I'm stationary in the office with both the hallway sensor (M5Stack) and Office ceiling light are right next to the office door (same horizontal distance but switch is about 30cm higher up. The bathroom heater switch is in the next room through an interior wall (but about same distance).

As the start to a spawning idea, in this case the average between ceiling light and hallway sensor should give a clearer signal. Might be interesting to see.

[agittins]

Hey, I was wondering if you had to do much work to set up your shelly's, or what settings you are using on them? On the community thread, lazmo88 is seeing a lot of big gaps in packet reception, so I'm wondering if its the script setup on the shelly side that's giving them grief. In particular, looking at hist_interval in the dump_devices output shows gaps upwards of a minute at times - have you seen anything similar with yours?

[agittins]

(oh, also - I'm stoked about the code you pulled together for trilateration and I'm itching to get it integration as soon as I can!)

[jaymunro]

Hey, I was wondering if you had to do much work to set up your shelly's, or what settings you are using on them? On the community thread, lazmo88 is seeing a lot of big gaps in packet reception, so I'm wondering if its the script setup on the shelly side that's giving them grief. In particular, looking at hist_interval in the dump_devices output shows gaps upwards of a minute at times - have you seen anything similar with yours?

https://community.home-assistant.io/t/bermuda-bluetooth-ble-room-presence-and-tracking-custom-integration/625780/84?u=hjm

That's all I did. Just 10 seconds beyond setting up the Shelly's on the WiFi and accepting the prompt to add the Shelly's inside HA. (Plus about 1-2 hours of investigation on the difference between the Shelly WebUI setting 'Enable Bluetooth gateway' and the HA setting 'Bluetooth scanner mode').

Once I figured it out though, adding a Shelly and setting it up as a "Bluetooth scanner" is just seconds per device. Additionally, they are super reliable and push updates immediately into HA. I have about 13 now and will be adding the new Shelly Mini PM into all power sockets just to monitor whole house power usage. That'll be about another 15 Shellys @ 12 euros each.

[jaymunro]

(oh, also - I'm stoked about the code you pulled together for trilateration and I'm itching to get it integration as soon as I can!)

Yep, me too Ashley. Might be starting some new work soon so free time may go down a bit till I get up to speed.

All the same, that code just took one late night to pull together. I was surprised how easy it was.