Optimize time between polling share2nightscout-bridge to reduce ingest lag #7231
Conversation
When collecting date from Dexcom via share2bridge-dexcom before we would poll at regular intervals. Dexcom transmitters / apps refresh every 5 minutes give/take the inaccuracy in the transmitters clock. If we poll every 5 minutes, we will randomly introduce up to 5 minutes of lag before we refresh. For example, if we poll 4:59s after dexcom refreshed we will face a 4:59s lag seeing the latest data arrive This patch introduces a refresh adjustment. We look at the timestamp of the most recent data item and aim to refresh 5 minutes plus a buffer after that timestamp. for example, 5:20s after the most recent data time. the additional buffer gives dexcom a chance to push the data through their application/infrastructure (after the app sends the data, it appears in testing to be around 20s before it is available in the API) This reduces the lag time to a reliable 20 seconds regardless of when nightscout was started and keeps it in track as it drifts forwards and back.
The default interval is increasd from 2.5 minutes to 2.6 minutes This change incorporates the lag between the app uploading and the data being available in share2nightscout-bridge This change also accomodates the optimisation of the refresh interval based on the last collected data timestamp
|
Thank you! I will see about getting this into the pending release. |
|
@bewest No urgency! I've moved my deployment to dev in order to try and help with testing. I have been working on support for LaMetric Time LED clocks devices. The endpoint displays the current trend/bg and 3 hours of history in a graph.
I've been running these features in my local copy for about 18 months. Raising PRs here and tracking against dev is going to get me up to speed for help with testing the dev branch. This PR here is a result of me noticing the lag in the endpoint updating and working to reduce it so my clock more closely followed the app. It's been pretty stable so it will be awesome to have it merged |
…chronize_dexcom_fetch_times Optimize time between polling share2nightscout-bridge to reduce ingest lag
|
Love this addition, thank you! |
|
Hey, First of all i would like to thank you for this wonderful improvement. Everyone who deals with diabetes knows that time matters very often. To be honest I am not very familiar with the technical architecture. Actually I am using the bridge to get values from follower and push it to nightscout. I thought that this is a totally separate application and does not have any dependencies to nightscout beside the Api. I call the bridge as systemctl service and set its settings over env vars. Now I am confused that there is a strict integration into nightscout?! How does it work from technical perspective. Am I doing something wrong? Best regards Selçuk |
|
The bridge can be invoked directly, but nightscout typically treats it as a code module, invoking the methods to collect the Dexcom data on its own timer Running as a stand alone process does a similar job, but the bridge manages the timing. This PR I raised alters the timer part in nightscout (looking at the time stamp of the most recently collected reading to schedule the next collection 5mins after that) I didn't make the same change to the bridge's own timer code but it wouldn't be a stretch to implement it there too. So you shouldn't need to run the bridge independently, though that would work (provided the bridge process knows how to post the data back to the nightscout process) Let me know if I could help more |
|
Thank you for your answer. Is there some kind of documentation which describes how to setup/configure and run this? |
This updates the polling interval for the share2nightscout-bridge fetch routine.
Current behaviour
Where as previously the share2nightscout-bridge is called at a regular interval to poll Dexcom for new data, we now use the data we received to schedule the next poll. With a default poll time of 2.5 minutes, depending on when the nightscout process is started, we could see up to a 2.5 minute lag between the new data being sent by the Dexcom app and it appearing in the nightscout feed.
Optimisation
We know that Dexcom updates every 5 minutes. We also know that it takes under 20 seconds for the new data sent by the Dexcom App to appear in the Dexcom feed.
We look at the timestamp of the most recently collected data, add 5 minutes and 20 seconds to this time to find the next poll time.
If this time is in the past or if this poll time is more than the poll interval in the future, we revert to regular interval polling. This reverts to the previous behaviour whilst we await new data from Dexcom.
Once new data is provided, we attempt to re-schedule the next collection optimally.
For example:
Result expected
This update will reduce the random lag of up to 2.5 minutes to a reliable 20 seconds