Skip to content

Fix loop runs lost when calculation chain is replaced (#5066) - #5087

Closed
zibra wants to merge 10 commits into
nightscout:devfrom
zibra:fix/loop-trigger-lost-5066
Closed

Fix loop runs lost when calculation chain is replaced (#5066)#5087
zibra wants to merge 10 commits into
nightscout:devfrom
zibra:fix/loop-trigger-lost-5066

Conversation

@zibra

@zibra zibra commented Aug 18, 2026

Copy link
Copy Markdown

Problem

Fixes #5066, related to #3372.

The loop can stop running for minutes to hours while BG values keep coming every minute. The app itself stays alive: SMS commands, pump communication and notifications still work. Bringing the app to the foreground or restarting it helps.

Log analysis in #5066 (thanks @kameamea) shows the mechanism: with 1-minute CGM data (Libre 3 via Juggluco) the calculation chain is cancelled again and again by stopCalculation(... onEventNewHistoryData) before it reaches the worker that invokes the loop. In one 32-minute window: 32 new BG values, 67 stopCalculation calls, 66 PrepareBasalDataWorker ... FAILURE {Error : stopped}, 0 loop runs.

Root cause

The loop was invoked only when the finished calculation chain was started by a new BG (triggeredByNewBG=true). When a BG-triggered chain is cancelled and replaced by a chain started by another DB change (treatment, temp basal, profile switch from NS sync), the replacement chain carries triggeredByNewBG=false. The loop run for that BG is then lost. Loop.lastBgTriggeredRun is never updated, so nothing recovers it. With 5-minute CGM data this race is rare; with 1-minute data it can win for hours.

Three more silent failure points were found with the same symptom profile (loop dead, everything else alive, restart fixes it). Each one could stop all future calculations for the rest of the process lifetime.

Changes

1. Do not lose the loop trigger when a chain is replaced (workflow)

PostCalculationWorker now invokes the loop after any completed MAIN calculation when the newest BG was not used for a loop run yet. The existing timestamp check against Loop.lastBgTriggeredRun still prevents duplicate runs. The triggeredByNewBG field was removed from PostCalculationData because it is no longer needed there.

2. Fix silent failures in IobCobCalculatorPlugin (plugins:main)

  • An exception inside the debounced newHistoryData task was swallowed by the executor and scheduledData was never cleared, so no calculation was ever scheduled again. The task now logs the error and always clears the state in a finally block.
  • historyWorker was created after the DB observers were registered. A DB change arriving in that window caused the same permanent stop. The executor is now created first.
  • The flow collectors (GV, CA, BS, BCR, TB, EB, EPS, preferences, units) had no error handling, so one exception cancelled the collection forever. The GV collector is the only entry point for BG-triggered loop runs. A shared onEachSafe wrapper now logs the error and keeps collecting.

3. Loop watchdog in KeepAliveWorker (core:interfaces, implementation)

Safety net for any remaining case. Every KeepAlive tick (about 5 minutes): when the newest BG is fresh (under 9 minutes old) but more than 10 minutes newer than the last BG that triggered a loop run, force a full recalculation through the new IobCobCalculator.forceRecalculation(). It starts the calculation directly and bypasses the debounce path, so it also works when that path is stuck. A LoopWatchdogFired analytics event keeps remaining cases visible.

Tests

  • PostCalculationWorkerTest: 2 new cases (skip loop when newest BG already used, skip when no actual BG).
  • KeepAliveWorkerTest: 5 new watchdog cases (fires on lost trigger, skips on small gap, stale BG, app start, client mode).
  • :app:compileFullDebugKotlin and both test classes pass.

Runtime behavior is being verified on a device with 1-minute Libre 3 data where the problem reproduced.

🤖 Generated with Claude Code

Tomasz Przysiwek and others added 4 commits August 18, 2026 13:59
The loop was invoked only when the finished calculation chain was started
by a new BG. A chain started by a new BG can be cancelled and replaced by
a chain started by another DB change before it reaches
PostCalculationWorker. The replacement chain had triggeredByNewBG=false,
so the loop run for that BG was lost. With 1-minute CGM data (Libre 3)
this could repeat for hours and no loop run happened at all.

Now the loop is invoked after any completed MAIN calculation when the
newest BG was not used for a loop run yet. The existing timestamp check
against Loop.lastBgTriggeredRun still prevents duplicate runs.

Issue nightscout#5066, nightscout#3372

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three failure points could stop BG processing forever while the rest of
the app kept working (SMS, pump queue, notifications):

- An exception inside the debounced newHistoryData task was swallowed by
  the ScheduledExecutorService. scheduledData was never cleared, so no
  calculation was ever scheduled again. Now the task logs the error and
  always clears the state in a finally block.
- historyWorker was created after the DB observers were registered. A DB
  change arriving in that window set scheduledData while nothing was
  scheduled, with the same permanent result. Now the executor is created
  first.
- The flow collectors (GV, CA, BS, BCR, TB, EB, EPS, preferences, units)
  had no error handling. One exception cancelled the collection forever.
  The GV collector is the only entry point for BG-triggered loop runs.
  Now a shared onEachSafe wrapper logs the error and keeps collecting.

Issue nightscout#5066, nightscout#3372

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Safety net for any remaining case where the calculation chain dies while
BG values keep coming. Every KeepAlive tick (about 5 minutes) checks:
when the newest BG is fresh (under 9 minutes old) but is more than 10
minutes newer than the last BG that triggered a loop run, force a full
recalculation.

The new IobCobCalculator.forceRecalculation() starts the calculation
directly and bypasses the debounce path, so it also works when that path
is stuck. The forced calculation ends with a loop run because the newest
BG was not used for a loop run yet.

Fires a LoopWatchdogFired analytics event so remaining cases stay
visible.

Issue nightscout#5066, nightscout#3372

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@kameamea

Copy link
Copy Markdown

Thanks @zibra for providing those fixes. I wonder whether almost no one realizes the severity of this problem, or why is this not included into dev with high priority? @MilosKozak
With the Libre sensor I have no AAPS loop for hours every day with all the bad consequences. IMHO a warning and/or fix should go out to all libre users.
I appreciate very much the effort you all put into this great software, and I know that time resources are limited, but this problem is really serious, I'd say potentially life threatening.
Sorry for crying for help this way, I don't know whom to address.

@JetFoxy

JetFoxy commented Sep 2, 2026

Copy link
Copy Markdown

I hit the same user-visible symptom as #5066 on a fork tracking dev (59ace5777a, 2026-08-13), also with 1-minute Libre data via Juggluco, and captured a log of it. I opened #5101 for it before finding this PR. Having now read the diff and re-measured my log against it, I think the two are the same root cause seen at two different stages of the pipeline, and that this PR is necessary but not sufficient. Posting the measurements here since they bear directly on whether the watchdog in change 3 can recover the case.

(Disclosure per CONTRIBUTING.md: the log correlation and code tracing below were done with AI assistance. I am deliberately not proposing a patch — everything below is verifiable from the repo and the log.)

What my log shows

77-minute window, BG arriving every ~60 s, 78 Running newHistoryData cycles. Throughout the whole window the autosens store never advances:

lastData=30.08.2026 09:18:00     <- 22 separate samples, spanning 09:43 to 11:01

AUTOSENSDATA null: data is old (Overview) size()=4605 lastData=30.08.2026 09:18:00

Zero APSResult inserted in those 77 minutes. So in my case the loop is not merely missing its trigger — there is no fresh autosens data for it to act on at all. That is the part change 1 cannot reach: PostCalculationWorker.invokeLoop() would still find iobCobCalculator.ads.actualBg() stale (pinned at 09:18) and return at the same place it does today.

Why the calculation never lands

Per-bucket cost on this device is ~25 s:

10:59:49.320  Processing calculation thread: NewBG (22/408)
11:00:14.027  Processing calculation thread: NewBG (21/408)
11:00:15.253  >>> IobCobOref1Thread <<< executed in 51492 milliseconds

Dominated by SensitivityOref1Plugin.detectSensitivity(), which is called once per bucket from runIobCobOref1() and each time walks the entire autosensDataTable (while (index < ads.autosensDataTable.size())) twice — once per 8 h/24 h segment — appending to a pastSensitivity StringBuilder. My table is at 4605 entries (Records: 4606) and is never pruned (Removing from autosensDataTable appears 0 times in the log). So one pass is O(buckets × table), and ~21 buckets are needed to reach "now" while only ~2 fit in a 60 s BG interval.

The part that surprised me, and that affects change 3

I had assumed the runs were being cancelled and discarding their work. They mostly are not — only 5 of ~78 cycles took the isStopped path (all between 09:53 and 10:18; none in the last 40 minutes). The rest ran the loop to completion and executed data.iobCobCalculator.ads = ads.

What they publish, though, is a clone taken ~50 s earlier — and stopCalculation() has already returned by then:

11:00:14.770  Stopping calculation thread: onEventNewHistoryData
11:00:14.776  Calculation thread stopped: onEventNewHistoryData     <- 6 ms later
11:00:14.776  Invalidating cached data to: 30.08.2026 10:55:00
11:00:14.776  Starting calculation worker: NewBG to 30.08.2026 11:00:14
11:00:15.253  AUTOSENSDATA thread ended: NewBG                      <- worker was still running

Same shape at 11:01:14 (+26 ms) and 11:01:24 (+2.3 s). stopCalculation() polls getWorkInfosForUniqueWork(job).get() for state != RUNNING, but cancelUniqueWork flips the WorkInfo to CANCELLED essentially immediately, while the coroutine keeps going until its next cooperative isStopped check — which inside runIobCobOref1 sits at the top of the bucket loop, i.e. up to one full bucket (~25 s here) away. So stopCalculation reports success while the old worker is very much alive, the caller then invalidates the shared ads and enqueues a new chain, and the old worker's iobCobCalculator.ads = ads lands afterwards — a lost update that reverts the invalidation.

This is why I think change 3 cannot recover my case. forceRecalculation() is documented as:

// No stopCalculation and no cache invalidation needed: history did not change,
// and the REPLACE policy inside runCalculation cancels a running chain anyway.

That holds for WorkManager's bookkeeping but not for the coroutine: the in-flight worker is not actually stopped at that instant, and it will publish its stale clone over whatever the forced chain sets up. The watchdog's precondition is also exactly satisfied in this state — lastBgTriggeredRun never advances (no chain ever completes with fresh data), newest BG stays fresh — so it would fire every KeepAlive tick and add one more participant to the same race every ~5 minutes.

One more thing that looks independently wrong

AutosensDataStoreObject.clone() copies bgReadings, autosensDataTable and bucketedData, but not referenceTime:

override fun clone(): AutosensDataStore =
    AutosensDataStoreObject().also {
        synchronized(dataLock) {
            it.bgReadings = this.bgReadings.toMutableList()
            it.autosensDataTable = LongSparseArray<AutosensData>(...).apply { putAll(...) }
            it.bucketedData = this.bucketedData?.toMutableList()
        }
    }

Since runIobCobOref1 publishes that clone back into iobCobCalculator.ads, the live store ends each cycle with referenceTime == -1. The next loadBgData()createBucketedDataRecalculated()adjustToReferenceTime() then re-anchors the whole 5-minute bucket grid to bgReadings[0].timestamp, i.e. to whatever the newest BG happens to be. On a 5-minute source that is harmless (and createBucketedData5min never calls adjustToReferenceTime at all); on a 1-minute source the anchor moves with every reading, which would shift bucket timestamps and miss every cached autosensDataTable key. I could not fully separate this effect from the lost update above using only the logs I have, so I am flagging it as "looks wrong on reading" rather than claiming it is the driver — but it would produce exactly the observed "recomputes the same wall-clock buckets forever" behaviour, and it is another instance of state that does not survive the publish boundary.

Why this reads as one root cause rather than four bugs

Every one of these is the same structural thing: the pipeline uses cancel-and-replace as its only concurrency primitive, and every piece of state it needs lives inside the run that gets replaced. What gets dropped at that boundary differs:

  • the loop trigger (triggeredByNewBG bound to the chain) — change 1 here
  • the scheduler state (scheduledData never cleared, collectors dying) — change 2 here
  • the bucket-grid anchor (referenceTime not cloned)
  • the freshly-computed autosens table (published from a clone taken before an invalidation that has since happened, because stopCalculation returned early)

It stays latent while a full pass is short relative to the input interval. 1-minute CGM breaks that assumption from both sides at once — 5× more restarts, and (via isAbout5minData) the more expensive recalculated bucketing path — which is why the reports cluster on Libre/Juggluco users.

Happy to test any candidate on the same setup and report back, or to attach the full logs if useful. Details and file/line references are in #5101, which I will correct to match these measurements (my original framing there over-weighted the cancellation path).

Tomasz Przysiwek and others added 2 commits September 2, 2026 12:33
A cancelled calculation worker keeps running until its next isStopped
check, up to one full bucket (tens of seconds on slow devices with a
lot of data). stopCalculation() returns as soon as WorkManager marks
the work CANCELLED, so the caller invalidates the store and starts a
new chain while the old worker is still alive. The old worker then
published its stale clone over the new state (lost update).

Re-check the chain generation right before the publish and skip it
when the worker was replaced. Also correct the comment on
forceRecalculation() which claimed the running chain is cancelled
immediately.

Based on log analysis by JetFoxy in PR nightscout#5087 / issue nightscout#5101.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014wDqc1Bv4AWZ7hcCrQEBx6
clone() did not copy referenceTime, and the calculation publishes the
clone back as the live store. The live store then lost its bucket grid
anchor after every calculation. On the next load the grid was
re-anchored to the newest BG. With a 1-minute CGM source the anchor
moved with every reading, so all bucket timestamps shifted and every
cached autosensDataTable entry became unreachable. The calculation
then recomputed the same wall-clock buckets forever and never reached
'now' (issue nightscout#5066, nightscout#5101).

Found by JetFoxy in PR nightscout#5087 review.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014wDqc1Bv4AWZ7hcCrQEBx6
@zibra

zibra commented Sep 2, 2026

Copy link
Copy Markdown
Author

@JetFoxy thank you very much for the detailed log analysis and code tracing — this is exactly the kind of input that moves a fix like this forward. I checked all four points against the code and they are correct. I pushed three new commits based on them:

1. Stale publish guard (9f2e635) — PrepareGraphDataWorker now re-checks the chain generation (via the existing WorkflowChainData generation counter) right before data.iobCobCalculator.ads = ads and skips the publish when the worker was replaced. This closes the lost update you measured: stopCalculation() returning while the cancelled coroutine is still up to one bucket (~25 s on your device) away from its next isStopped check. The comment on forceRecalculation() that wrongly claimed the running chain is cancelled immediately is corrected too.

2. referenceTime survives clone() (c026ba3) — one line plus a test. As you spotted, the calculation publishes the clone back as the live store, so the bucket grid anchor was lost after every cycle and re-anchored to the newest BG on the next load. On a 1-minute source that shifts all bucket timestamps every reading and makes every cached autosensDataTable entry unreachable — which matches your "table pinned at 09:18, same buckets recomputed forever" observation. This looks like the main driver of your case.

3. Watchdog rate limit (15d981b) — the KeepAlive loop watchdog now fires at most once per 15 minutes, so it cannot add a new cancel-and-replace participant every 5 minutes while a slow forced recalculation is still running.

Out of scope for this PR, left for #5101: the O(buckets × table) cost of detectSensitivity(), pruning of autosensDataTable, and a real redesign of stopCalculation() so it waits for the coroutine and not only for the WorkManager state. I agree with your framing that these are all the same structural problem (cancel-and-replace as the only concurrency primitive); the commits here remove the data loss at the publish boundary without that bigger redesign.

I will be testing now this version with real devices.

When the calculation never completes (for example the state fixed in
the previous commits), the watchdog precondition stays true on every
KeepAlive tick. Firing every 5 minutes would only add more
cancel-and-replace churn while a slow forced recalculation is still
running. Fire at most once per 15 minutes.

Suggested by JetFoxy in PR nightscout#5087 review.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014wDqc1Bv4AWZ7hcCrQEBx6
@sonarqubecloud

sonarqubecloud Bot commented Sep 2, 2026

Copy link
Copy Markdown

@JetFoxy

JetFoxy commented Sep 2, 2026

Copy link
Copy Markdown

Thanks for turning that around so fast — I read all three commits and they match what I measured. Two follow-ups, one of which I think matters before this is considered done.

The publish guard trades one all-or-nothing behaviour for another

publishAds() is the right call for the lost update, and you already flag the residual window in the comment. The part I'd want to think about is what a superseded worker now does with the buckets it did compute: it drops all of them.

In the steady state that is harmless — with referenceTime preserved the cache works, a pass only has to compute the handful of newly-invalidated buckets, finishes well inside the BG interval, and is never superseded. But in a genuine catch-up state (phone off overnight, a long BG gap, or the degraded state in the next section) a pass has to compute hundreds of buckets and cannot finish within one BG interval. Every pass is then superseded, so with this commit every pass publishes nothing, and the store stays pinned exactly as it did before — only now the reason is a deliberate skip rather than an overwrite.

So for that scenario the guard neither fixes nor worsens the outcome, it just changes the mechanism. What would actually break the deadlock is making progress survive supersession — e.g. merging the computed buckets into the live store instead of replacing it wholesale. The buckets a catch-up pass computes are hours in the past; the invalidation that superseded it only removed entries newer than newBG - 5 min, so the two do not overlap and a merge would not resurrect anything stale. That is a bigger change than this PR should carry, but it may be worth a note in the code or in #5101 so the remaining case is not lost.

The new Skipping ads publish (superseded) debug line is very welcome regardless — it makes this state identifiable in a log, which it was not before.

The unbounded autosensDataTable is probably what pushes instances over the threshold

This is the piece I under-weighted in my first comment, and I now think it explains the "restart fixes it, then it comes back days later" pattern that runs through #5066 and #5016.

autosensDataTable only ever loses entries in two places: newHistoryData(), which removes entries newer than the invalidation point and breaks on the first older one, and reset(), which is a full clear reached only from resetDataAndRunCalculation (config/preference change). Nothing removes entries that have aged out of the detection window. In my log the table was at 4605 entries against a ~34 h detection window, and Removing from autosensDataTable appears 0 times across the whole capture.

detectSensitivity() then iterates the entire table on every call (while (index < ads.autosensDataTable.size())), twice per call for the 8 h and 24 h segments, continue-ing past everything outside fromTime..toTime. Since it is called once per bucket, per-bucket cost scales with total table size rather than with the window — which is how I ended up at ~25 s per bucket.

That gives a clean explanation for the uptime correlation: a freshly started app has a small table, so passes are fast, they finish inside the BG interval, and everything works. The table grows monotonically with uptime, per-bucket cost grows with it, and at some point a pass can no longer finish within one BG interval — at which point the app is in the state above and only a restart (which clears the table) recovers it. Pruning the table to the detection window would cap the cost and raise that threshold a long way, and it looks like a contained change.

Not asking you to put it in this PR — you already scoped it to #5101 and I agree — but I wanted to record why I think it is the one that determines whether users hit this at all, rather than a general performance nit.

Testing

I'll build your branch and run it on the same device/setup (1-minute Libre via Juggluco, the one that produced the pinned-at-09:18 log) and report back what the logs look like — in particular whether the store now advances, and whether Skipping ads publish (superseded) shows up in normal operation. Will post results here.

Tomasz Przysiwek added 2 commits September 3, 2026 11:09
…5066

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012uQfwAT4RUr4Y27MBRDJ6K

# Conflicts:
#	core/keys/src/androidMain/res/values/strings.xml
#	core/keys/src/main/kotlin/app/aaps/core/keys/BooleanKey.kt
#	implementation/src/androidHostTest/kotlin/app/aaps/implementation/queue/CommandQueueImplementationTest.kt
#	implementation/src/androidMain/kotlin/app/aaps/implementation/receivers/KeepAliveWorker.kt
#	implementation/src/commonMain/kotlin/app/aaps/implementation/queue/CommandQueueImplementation.kt
#	plugins/main/src/commonMain/kotlin/app/aaps/plugins/main/iob/iobCobCalculator/IobCobCalculatorPlugin.kt
#	plugins/main/src/commonMain/kotlin/app/aaps/plugins/main/iob/iobCobCalculator/data/AutosensDataStoreObject.kt
#	workflow/src/androidHostTest/kotlin/app/aaps/workflow/PostCalculationRunnerTest.kt
#	workflow/src/commonMain/kotlin/app/aaps/workflow/PostCalculationRunner.kt
#	workflow/src/commonMain/kotlin/app/aaps/workflow/PrepareGraphDataRunner.kt
#	workflow/src/main/kotlin/app/aaps/workflow/CalculationWorkflowImpl.kt
@MilosKozak

Copy link
Copy Markdown
Contributor

I pushed 17dd2bb to dev. It is based on this PR and on @JetFoxy's analysis in this thread — thank you both, the log correlation here is what made the root cause findable.

The commit takes four of the six changes and leaves the watchdog out. Below is what is different and why, so you can see where I disagreed instead of having to diff it.

Taken as proposed

  • The loop trigger fix. PostCalculationRunner.invokeLoop() no longer asks whether the chain was started by a new BG. Loop.lastBgTriggeredRun is the only gate now, and PostCalculationData.triggeredByNewBG is gone. LoopPlugin.lastBgTriggeredRun also became @Volatile, because it is now the only thing standing between one loop run and two.
  • collectResilient for the nine remaining DB and preference collectors.

Taken with changes

1. referenceTime needs two more guards, or it trades one bug for another.

Once the anchor lives for the whole process, a new sensor can start on a different 5 minute phase. createBucketedData5min() then shifts the oldest reading, its own 90 second check trips, and every later run falls back to the recalculated path — permanently, and with the "recalculated data used" warning showing. dev now re-anchors when the anchor is more than 90 seconds out of phase, and adds the anchor shift back into that 90 second test, so the test measures jitter in the data again and not where the grid happens to sit.

2. publishAds() should skip only the store write, not EventAutosensCalculationFinished.

That event only asks its listeners to refresh, and the graph data in data.cache was written by this run either way. PersistentNotificationPlugin, the Android Auto merge and two instrumented tests (LoopTest, CobExtendedCarbsTest) wait on it.

3. The debounce body needs three more things next to the finally.

  • clearCache() and ads.reset() in the catch arm. Without them the invalidation stopped half way and the loop keeps dosing from a cache built on the old data.
  • The finally must clear only what this run owns (if (scheduledData === data)). A newer request can take the lock while our delay continuation is being dispatched, and by then it has published its own scheduledData and job.
  • scheduledData must also be cleared in onStop(), and must not be published at all when scope is null. Both leave an entry with no job behind it, which wedges the guard in the same way.

One more thing, which this PR cannot fix from inside the withLock

historyLock was an AapsLock, and on the JVM that is a ReentrantLock. The debounce body holds it across newHistoryData(), which suspends — WorkManagerCalculationExecutor.stop() polls with delay — on aapsIoDispatcher, which is Dispatchers.IO. If the coroutine resumes on another pool thread, the inline withLock calls unlock() from a thread that does not own the lock: IllegalMonitorStateException, the lock stays held, and every later scheduleHistoryDataChange blocks for the rest of the process.

This is a regression from the multiplatform move. Before it the same body ran on a single thread executor under synchronized, where the lock could not change threads. The try/finally in this PR sits inside that same withLock, so it does not close it. historyLock is now a kotlinx.coroutines.sync.Mutex and scheduleHistoryDataChange is suspend.

I have not seen this in a log, so I do not claim it is what users are hitting. It is a landmine in code this PR is already editing.

Not taken: the KeepAlive watchdog

I left checkLoopWatchdog() and IobCobCalculator.forceRecalculation() out for now:

  • forceRecalculation() does no stopCalculation and relies on the REPLACE policy, which is the cancel and replace this issue is about.
  • There is no in-flight check, so on a slow device it restarts a calculation that was going to finish, every 15 minutes.
  • lastBgTriggeredRun == 0L disarms it for good if the startup chain is the one that died.
  • LOOP_WATCHDOG_MAX_BG_AGE compares a raw database timestamp against the window of actualBg(), which is on the bucketed value.

If we still want a net after the rest is proven in the field, I would rather have a detection only version (log plus analytics, no action) in PeriodicMaintenance, so iOS gets it too.

Two behaviour changes worth knowing

  • The loop now runs once at app start, because lastBgTriggeredRun is 0 and is not persisted. It is still capped at one run per BG.
  • With a stable anchor the newest bucket for a 1 minute source moves in 5 minute steps, so those users go from about one loop run per minute to one per five minutes. That is the intended design, but it is a real change.

Still open

@JetFoxy is right that the unpruned autosensDataTable is what decides whether a user hits this at all: detectSensitivity() walks the whole table once per bucket. Nothing above caps that cost. There is also a second trigger per reading — the NS id write back calls updateExistingEntry, which feeds changeFlow again. Both belong in #5101.

Please also split the AlertFailedUpdateBasalProfileSound commit into its own PR. It is unrelated to this one.

Testing

I have compile and unit tests only (:app:compileFullDebugKotlin, testAndroidHostTest for :workflow, :plugins:main, :plugins:aps, :implementation, and compileKotlinIosArm64). You both have the setup that reproduces this, which I do not. Could you run dev and report whether the autosens store now advances, and whether Skipping ads publish (superseded) or Reference time out of phase with current data appear in normal use?

@JetFoxy

JetFoxy commented Sep 9, 2026

Copy link
Copy Markdown

Thanks for the detailed writeup of where you disagreed — that was much more useful than a diff would have been, and the two extra referenceTime guards in particular are a case I had not thought through: I only checked that the anchor stops moving, not what happens when a new sensor starts on a different phase.

I have moved the two remaining items into #5101 and re-scoped it to just those, with the original root-cause writeup folded into a details block for context. While writing it up I traced the NS id write-back myself to make sure I was describing it correctly, and it is exactly as you said:

  • DelegatedGlucoseValueDao.updateExistingEntry() adds to the same changes list that insertNewEntry() does, so observeChanges(GV::class) cannot tell a metadata-only update from a new reading.
  • UpdateNsIdGlucoseValueTransaction guards on current.interfaceIDs.nightscoutId != glucoseValue.interfaceIDs.nightscoutId, so it fires once per reading rather than repeatedly — a second full invalidate-and-restart per reading, carrying no new glucose data.
  • The same shape applies to the other UpdateNsId*Transactions whose entities feed collectors in IobCobCalculatorPlugin (Bolus, Carbs, TemporaryBasal, ExtendedBolus, BolusCalculatorResult), so whatever distinguishes a metadata update from a data change would cover all of them at once.

On the uncapped detectSensitivity() cost I have nothing to add beyond what is now in #5101 — the table only loses entries newer than the invalidation point or on a full reset(), so it grows with uptime while the cost is O(buckets × table).

On testing

One honest limitation: I cannot run stock dev on the device that produced the capture, because its pump driver is not upstream — the app would come up with no pump. So I cannot give you a clean "I ran dev" answer.

What I am doing instead is running your four changes ported onto the branch that produced the original log, on that same setup (1-minute Libre via Juggluco, Nightscout connected). The port is semantic rather than a cherry-pick, since that branch predates the multiplatform move and still has the workers and the single thread executor debounce — which also means the historyLock landmine you found does not exist there, so that part is untested by me either way. I will report:

  • whether the autosens store advances instead of pinning,
  • whether Skipping ads publish (superseded) shows up in normal operation, and how often,
  • whether Reference time out of phase with current data. Re-anchoring. appears outside a genuine sensor change,
  • and whether the loop-run rate on the 1-minute source really settles to one per five minutes as you expect.

That last one is the behaviour change I would most like to see confirmed on real data rather than reasoned about, given it is a fivefold drop in loop runs for exactly the users who reported the problem.

If a detection-only watchdog in PeriodicMaintenance does land later, I am happy to run that too — on this setup it would have fired continuously during the original incident, so it is a good negative control.

@JetFoxy

JetFoxy commented Sep 10, 2026

Copy link
Copy Markdown

One consequence of the loop-rate change that I think needs a decision before this reaches users, because it collides with a preference default.

The 5 minute floor also becomes an SMB floor

invokeLoop() gates on iobCobCalculator.ads.actualBg(), and that is the bucketed value, not the reading:

override fun actualBg(): InMemoryGlucoseValue? {
    val lastBg = lastBg() ?: return null          // lastBg() -> bucketedData[0]
    return if (lastBg.timestamp > System.currentTimeMillis() - T.mins(9).msecs()) lastBg else null
}

With the anchor now stable, bucketedData[0].timestamp is a grid point and advances in 5 minute steps, so lastBgTriggeredRun can only advance in 5 minute steps — this is exactly the "one per five minutes" you described. But an SMB is only ever issued from inside a loop run, so the same step becomes a floor on SMB frequency, whatever SMBInterval says.

ApsMaxSmbFrequency allows 1..10 minutes and defaults to 3:

ApsMaxSmbFrequency(
    key = "smbinterval",
    defaultValue = 3,
    min = 1,
    max = 10,

and the gate itself has only a 6 second tolerance, which assumes another attempt comes along soon:

val SMBInterval = min(10, max(1, profile.SMBInterval)) * 60.0
if (lastBolusAge > SMBInterval - 6.0) { ... } else { "Waiting ...s to microbolus again." }

So on a 1 minute source every value from 1 to 4 is now unreachable and silently behaves as 5. For the default that is SMB roughly every 5 minutes where it used to be every 3.

Before/after for a 1 minute source

The comparison is against a healthy pre-fix install, not against the stuck state:

  • Before, clone() dropped referenceTime, so createBucketedDataRecalculated() re-anchored to bgReadings[0].timestamp on every load. bucketedData[0] therefore tracked the newest reading and moved every minute, the loop ran every minute, and an SMBInterval of 3 was actually honoured.
  • After, the grid is fixed, so the floor is 5 minutes.

That is about 40% fewer SMB opportunities for anyone on a 1 minute source who left the default, in the same release that fixes their loop gaps. Running the loop more often than once per 5 minutes is a fairly common setup among these users, so this is not a corner case.

5 minute sources are unaffected: their readings advance the grid once per reading either way.

A second, weaker point

Even at SMBInterval = 5 the run-to-run spacing is exactly 300 s while lastBolusAge at the next run is 300 − δ, where δ is the delay between the decision and the bolus record. The 6 s tolerance is precisely the budget for δ, and if δ exceeds it the gate closes and the next chance is a full 5 minutes later, i.e. SMB every 10 minutes.

This one is probably fine as it stands: DetailedBolusInfo.timestamp is System.currentTimeMillis() at construction, before the pump round trip, so BLE latency does not enter the timestamp. It would only bite if that record is later replaced by a pump-history sync carrying a coarser timestamp. I have not seen it happen and am not claiming it does — flagging it because the margin is 6 seconds and nothing enforces it.

Unrelated, but it came out of the same check

The minute-spaced autosensDataTable rows in my earlier log (08:05, 08:06, 08:07, …) are not minute buckets — buckets are always 5 minutes (currentTime -= T.mins(5).msecs()). They are a fingerprint of the dropped anchor: each run used a grid shifted by one minute and added its own rows, so after five runs the table was populated at every minute.

That means referenceTime surviving clone() also cuts the growth rate of that table by roughly 5x, which takes some pressure off the uncapped detectSensitivity() cost in #5101. The 4605 entries I measured against a ~34 h window were largely this, not genuine data.

Options, as I see them

  1. Leave it, and document that on sub-5-minute sources SMBInterval has an effective floor of 5 minutes.
  2. Clamp or warn in the UI when smbinterval is below the achievable rate for the active source.
  3. Let the loop trigger track the reading rather than the bucket, e.g. gate invokeLoop() on the newest raw glucose value while keeping the bucketed value for the calculation itself. That restores the old cadence without touching the grid, but it puts the restart rate back up, which is the thing this PR is reducing — so it trades one problem for the other.

I do not have a recommendation I am confident in; the trade-off is clinical rather than technical. Happy to measure the actual SMB cadence on the 1 minute setup once I have the ported build running, which would at least turn "about 40% fewer" into a real number.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants