v2.10.0

Significant Changes

• Preferential ordering of streams during network recovery now allows certain streams to be pushed onto network quicker than others when recovering from a network incident, this can help users ensure that data on certain stream is not blocked due to large backlog on another stream. Configurable with the addition of a line into the config toml file as explained with the following example diff:

[streams.device_shadow]
topic = "/tenants/{tenant_id}/devices/{device_id}/events/device_shadow/jsonarray"
flush_period = 5
+ priority = 75

[streams.C2C_CAN]
topic = "/tenants/{tenant_id}/devices/{device_id}/events/C2C_CAN/jsonarray/lz4"
buf_size = 100
compression = "Lz4"
+ priority = 50

NOTE: Higher priority streams get pushed first, so in the above case the C2C_CAN stream will wait for data waiting on the device_shadow stream. Please ensure that device configurations are updated such that streams containing more important real-time data gets pushed earlier than others. We have already configured the action_status stream to have the highest priority given it's importance to the working of the platform, the default config of which is now as follows:

[action_status]
topic = "/tenants/{tenant_id}/devices/{device_id}/action/status"
buf_size = 1
flush_period = 2
priority = 255 # highest priority for quick delivery of action status info to platform

• New metrics stream uplink_serializer_stream_metrics tracks per stream serialization and compression stats(sizes and times).

NOTE: this is different from uplink_serializer_metrics which tracks the serializer as a whole and uplink_stream_metrics which tracks per stream stats on the data bridge.

• Uplink now ensures full graceful shutdown with all data on streams having disk persistence enabled being written onto disk during shutdown by force flushing their write buffers instead of waiting to overflow and hence getting dropped with the process exit. Similarly packets inflight/waiting for acknowledgement from broker are also saved onto disk and recovered during a restart event.

What's Changed

• feat: track disk space utilized by persistence by @de-sh in #312
• refactor: use StreamConfig in Serializer by @de-sh in #296
• refactor: handle files on disk as PersistenceFile by @de-sh in #314
• chore: use workspace dependencies by @de-sh in #315
• refactor: CtrlTx to send control messages to various uplink components by @de-sh in #316
• feat: enable simulation on non-demo tenants by @pranavlpin in #317
• feat: extract out system stat collector by @de-sh in #318
• fix: ensure storages are force flushed during uplink shutdown by @de-sh in #311
• feat: persist inflight publishes and send on restart by @de-sh in #299
• feat: track stream-wise compression stats in serializer by @de-sh in #320
• feat: preferential ordering of streams during read from disk by @de-sh in #289

New Contributors

• @pranavlpin made their first contribution in #317

Full Changelog: v2.9.1...v2.10.0