Prune old rows in `device_lists_changes_in_room` table. by erikjohnston · Pull Request #19473 · element-hq/synapse

erikjohnston · 2026-02-17T11:12:36Z

The usages of the table mostly already correctly handled if we don't have old entries, as that was needed when we first added the table.

I arbitrarily set the prune time to 30 days. The only use for old entries is for sync streams that haven't synced since then, and we should very rarely see sync streams that haven't been used in 30 days.

Reviewable commit-by-commit.

We called `txn.rowcount` *after* we used the txn for something else, so we were no longer counting rows deleted by the prune but rows inserted into the cache stream. This caused a tight loop.

reivilibre

Structurally sane :)

reivilibre · 2026-02-27T13:15:09Z

+            return num_deleted
+
+        num_rows_deleted = 0
+        while True:


Are we happy that there isn't much in the way of rate control on these deletions?

Co-authored-by: Olivier 'reivilibre' <oliverw@element.io>

…evices_in_rooms

Co-authored-by: Olivier 'reivilibre' <olivier@librepush.net>

reivilibre

Seems like it's basically there, save for a bit of potential raciness and maybe a test?

reivilibre · 2026-04-09T17:58:57Z

+        if changes is not None:
+            local_changes = {(u, d) for u, d in changes if self.hs.is_mine_id(u)}
+        else:
+            # The `device_lists_stream_id` is too old, so we need to fall back


how much of a pain would it be to stand up a test against this case? I worry this is probably untested code. Being an edge case, it will therefore probably go unnoticed if it breaks until a very awkward moment. Would be nice to have something

Right, managed to wrangle a test bb6c559

Co-authored-by: Olivier 'reivilibre' <oliverw@element.io>

Instead of checking it in a separate transaction, check in the transaction we're reading the table from.

Rather than trying to infer it from the minimum ID in the table. We were seeing issues in CI due to not all device list updates having entries in the `device_lists_changes_in_rooms` table due to the user not being in any rooms. This meant that the returned minimum ID was larger than expected, causing failures when calling `get_all_device_list_changes(..)`. By explicitly tracking the max pruned ID, we don't have to worry about problems with trying to infer the actual max pruned ID.

erikjohnston · 2026-04-14T15:03:31Z

@reivilibre sorry, the worker integration tests raised an issue around the fact that taking the minimum stream ID of device_lists_changes_in_room is not a reliable way of determining the minimum safe stream ID. fee4c3f changes things so that we explicitly track in a dedicated table the maximum stream ID that we have pruned

reivilibre

Probably sane but would be good to have a really good idea of what the new table is for; it does feel like the kind of thing someone will dig up years down the line and have to scratch their head over it.

It doesn't sound insane so probably just a matter of getting the illustration right, e.g. with a step by step sequence to show what goes wrong with the 'simple' MIN(stream_id)

reivilibre · 2026-04-14T16:18:01Z

+-- the table cannot provide a complete answer.
+--
+-- This replaces the previous approach of using MIN(stream_id) on the
+-- device_lists_changes_in_room table, which incorrectly returned 0 when


Ironically this new table is more confusing to me, as in I don't see why this is preventing anything.

By my reading, COALESCE(MIN(stream_id), 0) is the reason the previous approach returned 0 when empty. (So I think the comment here is maybe a bit misleading, or doesn't quite reveal the reason we need this?)

Trying to understand the difference, here is a (step by step) comparison table as I see it:

Old _get_min_device_lists_changes_in_room New _get_max_pruned_device_lists_changes_in_room_txn

Seed (empty table) returns 0 returns 0

New rows added with stream_ids 2..=15 (empty table) returns 2 still returns 0

Prune with prune_before_stream_id 10 now returns MIN(stream_id) = 10 now returns 10 (explicitly stored)

Old _get_min_device_lists_changes_in_room New _get_max_pruned_device_lists_changes_in_room_txn

Seed (populated table with stream_ids 2..=9) returns MIN(stream_id) = 2 returns MIN(stream_id) - 1 = 1

New rows added with stream_ids 10..=15 (populated table) still returns 2 still returns 1

Prune with prune_before_stream_id 10 now returns MIN(stream_id) = 10 now returns 10 (explicitly stored)

Apart from the initial state before the first prune, it looks like both approaches behave the same between prunes?
What am I missing? Maybe the device_lists_changes_in_room rows get deleted as rooms get purged?

By my reading, COALESCE(MIN(stream_id), 0) is the reason the previous approach returned 0 when empty. (So I think the comment here is maybe a bit misleading, or doesn't quite reveal the reason we need this?)

Argh, sorry that comment should have been deleted. I was messing around with testing getting LLM to generate the patch and it misunderstood the rationale (but got the change correct). I changed it locally but it got swallowed.

The actual problem (as per commit comment) comes from when we generate stream positions that don't have associated data in the device_lists_changes_in_rooms table, e.g. because the user isn't in any rooms. In that case if we insert a row into device_lists_stream table but not in device_lists_changes_in_rooms, and then later one that inserts rows into both, then MIN(stream_id) will return a stream ID greater than the first row even though its data hasn't been pruned (there was just no associated data to fetch).

reivilibre

LGTM otherwise, thanks!

reivilibre · 2026-04-16T10:04:20Z

+-- it's safe to read from that table for a given stream_id — if the
+-- requested stream_id is < the value here, the data has been pruned and
+-- the table cannot provide a complete answer.
+CREATE TABLE IF NOT EXISTS device_lists_changes_in_room_max_pruned_stream_id (


would be good to still have a little distilled comment about why we need this table, which is that device_lists_stream somehow ties into this (I don't have a great picture of what this is otherwise I'd suggest some wording)

Does e49192d help?

Follows on from #19473. We should be recording where we have deleted up to in the same transaction as we perform the delete, rather than at the end. Also let's log more regularly, as the initial set of deletions will likely take a long time

Follows on from #19473. We should be recording where we have deleted up to in the same transaction as we perform the delete, rather than at the end. This code only starts deleting rows after a month (and the original PR isn't in a release yet), so no server should have run into this problem yet. Also let's log more regularly, as the initial set of deletions will likely take a long time.

…9473) Fixes element-hq#13043 The usages of the table mostly already correctly handled if we don't have old entries, as that was needed when we first added the table. I arbitrarily set the prune time to 30 days. The only use for old entries is for sync streams that haven't synced since then, and we should very rarely see sync streams that haven't been used in 30 days. Reviewable commit-by-commit. --------- Co-authored-by: Olivier 'reivilibre' <oliverw@element.io> Co-authored-by: Olivier 'reivilibre' <olivier@librepush.net>

Follows on from element-hq#19473. We should be recording where we have deleted up to in the same transaction as we perform the delete, rather than at the end. This code only starts deleting rows after a month (and the original PR isn't in a release yet), so no server should have run into this problem yet. Also let's log more regularly, as the initial set of deletions will likely take a long time.

Taken and adapted from #19473 Didn't add the "safety" table due to afaik that has more relevance with the device updates, and not so much here for the profile updates.

@jason-famedly

Tested on NetBSD 10 amd64 with 2026Q2 environment. # Synapse 1.156.0 (2026-07-07) ## Features - Expose [MSC4354 Sticky Events](matrix-org/matrix-spec-proposals#4354) over [MSC4186 (Simplified) Sliding Sync](matrix-org/matrix-spec-proposals#4186). ([\#19591](element-hq/synapse#19591)) - Stabilize support for sending ephemeral events to application services, as per [MSC2409](matrix-org/matrix-spec-proposals#2409). Contributed by @jason-famedly @ Famedly. ([\#19758](element-hq/synapse#19758)) - Include `allowed_room_ids` in the `/summary` client-server API response for rooms with restricted join rules, as required by Matrix 1.15. Contributed by @FrenchGithubUser @famedly. ([\#19762](element-hq/synapse#19762)) - [MSC4140: Cancellable delayed events](matrix-org/matrix-spec-proposals#4140): Allow authentication on delayed event management endpoints (such as `/restart`) to bypass ratelimits for unauthenticated requests based on the client IP address. ([\#19794](element-hq/synapse#19794)) - Add new metric `synapse_non_deactivated_user_count` which tracks the number of non-deactivated users in the database, split by `app_service`. ([\#19848](element-hq/synapse#19848)) - The `GET /_matrix/client/unstable/org.matrix.msc1763/retention/configuration` endpoint is now provided when retention is enabled and `experimental_features.msc1763_enabled` is enabled, based on [MSC1763](matrix-org/matrix-spec-proposals#1763). ([\#19853](element-hq/synapse#19853)) - Add experimental support for [MSC4491: Invite reasons in room creation](matrix-org/matrix-spec-proposals#4491). ([\#19874](element-hq/synapse#19874)) # Synapse 1.155.0 (2026-06-16) # Synapse 1.154.0 (2026-06-04) ## Features - Add support for [MSC4452: Preview URL capabilities API](matrix-org/matrix-spec-proposals#4452) which exposes a `io.element.msc4452.preview_url` capability. If `experimental_features.msc4452_enabled` is `true`, the `/_matrix/(client/v1/media|media/v3)/preview_url` endpoint now responds with a 403 status code when the capability is disabled. ([\#19715](element-hq/synapse#19715)) # Synapse 1.153.0 (2026-05-19) ## Features - Make ACLs apply to EDUs per [MSC4163](matrix-org/matrix-spec-proposals#4163). ([\#18475](element-hq/synapse#18475)) - Stabilize [MSC3266: Room summary API](matrix-org/matrix-spec-proposals#3266), removing the experimental config flag `msc3266_enabled`. Contributed by @dasha-uwu. ([\#19720](element-hq/synapse#19720)) - Partial [MSC4311](matrix-org/matrix-spec-proposals#4311) implementation: `m.room.create` is now a required part of stripped `invite_state`/`knock_state` . Contributed by @FrenchGithubUser @famedly. ([\#19722](element-hq/synapse#19722)) - Expose `tombstoned` and `replacement_room` in room details on admin API endpoint `GET /_synapse/admin/v1/rooms/<room_id>`. Contributed by Noah Markert. ([\#19737](element-hq/synapse#19737)) # Synapse 1.152.1 (2026-05-07) # Synapse 1.152.0 (2026-04-28) ## Features - Add a ["Listing quarantined media changes" Admin API](https://element-hq.github.io/synapse/latest/admin_api/media_admin_api.html#listing-quarantined-media-changes) for retrieving a paginated record of when media became (un)quarantined. ([\#19558](element-hq/synapse#19558), [\#19677](element-hq/synapse#19677), [\#19694](element-hq/synapse#19694)) - Advertise [MSC4445](matrix-org/matrix-spec-proposals#4445) sync timeline order in `unstable_features`. ([\#19642](element-hq/synapse#19642)) - Report the Rust compiler version used in the Prometheus metrics. Contributed by Noah Markert. ([\#19643](element-hq/synapse#19643)) - Passthrough 'article' and 'profile' OpenGraph metadata on URL preview requests. ([\#19659](element-hq/synapse#19659)) - Add a way to re-sign local events with a new signing key. ([\#19668](element-hq/synapse#19668)) - Support [MSC4450: Identity Provider selection for User-Interactive Authentication with Legacy Single Sign-On](matrix-org/matrix-spec-proposals#4450). ([\#19693](element-hq/synapse#19693)) - Add experimental support for [MSC4242](matrix-org/matrix-spec-proposals#4242): State DAGs. Excludes federation support. ([\#19424](element-hq/synapse#19424)) - Adds [Admin API](https://element-hq.github.io/synapse/latest/usage/administration/admin_api/index.html) endpoints to list, fetch and delete user reports. ([\#19657](element-hq/synapse#19657)) - Reduce database disk space usage by pruning old rows from `device_lists_changes_in_room`. ([\#19473](element-hq/synapse#19473), [\#19709](element-hq/synapse#19709)) # Synapse 1.151.0 (2026-04-07) ## Features - Add stable support for [MSC4284](matrix-org/matrix-spec-proposals#4284) Policy Servers. ([\#19503](element-hq/synapse#19503)) - Update and stabilize support for [MSC2666](matrix-org/matrix-spec-proposals#2666): Get rooms in common with another user. Contributed by @tulir @ Beeper. ([\#19511](element-hq/synapse#19511)) - Updated experimental support for [MSC4388: Secure out-of-band channel for sign in with QR](matrix-org/matrix-spec-proposals#4388). ([\#19573](element-hq/synapse#19573)) - Stabilize `room_version` and `encryption` fields in the space/room `/hierarchy` API (part of [MSC3266](matrix-org/matrix-spec-proposals#3266)). ([\#19576](element-hq/synapse#19576)) - Introduce a [configuration option](https://element-hq.github.io/synapse/latest/usage/configuration/config_documentation.html#matrix_authentication_service) to allow using HTTP/2 over plaintext when Synapse connects to Matrix Authentication Service. ([\#19586](element-hq/synapse#19586)) ## Deprecations and Removals - Remove support for [MSC3852: Expose user agent information on Device](matrix-org/matrix-spec-proposals#3852) as the MSC was closed. ([\#19430](element-hq/synapse#19430)) # Synapse 1.150.0 (2026-03-24) ## Features - Add experimental support for the [MSC4370](matrix-org/matrix-spec-proposals#4370) Federation API `GET /extremities` endpoint. ([\#19314](element-hq/synapse#19314)) - [MSC4140: Cancellable delayed events](matrix-org/matrix-spec-proposals#4140): When persisting a delayed event to the timeline, include its `delay_id` in the event's `unsigned` section in `/sync` responses to the event sender. ([\#19479](element-hq/synapse#19479)) - Expose [MSC4354 Sticky Events](matrix-org/matrix-spec-proposals#4354) over the legacy (v3) /sync API. ([\#19487](element-hq/synapse#19487)) - When Matrix Authentication Service (MAS) integration is enabled, allow MAS to set the user locked status in Synapse. ([\#19554](element-hq/synapse#19554))

…saction gets retried. (#19947) Introduced in: #19473 Noticed in: #19556 (comment) I have not experienced the bug in the real world, it's just something I noticed by reading. -- Fix bug in `_prune_device_lists_changes_in_room` when transaction is retried The `nonlocal` variable is a footgun as it increments the counter even though the transaction did not commit yet and may still be retried. --------- Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>

erikjohnston added 6 commits February 16, 2026 14:37

Handle device list ID too old in get_device_list_changes_in_room

e945747

Add index

6883d48

Insert timestamp

155e941

Add background job to prune old device lists changes

dd612e0

Add test

0041c7e

Newsfile

fb88b76

erikjohnston marked this pull request as ready for review February 17, 2026 11:33

erikjohnston requested a review from a team as a code owner February 17, 2026 11:33

Fix tight loop

c517139

We called `txn.rowcount` *after* we used the txn for something else, so we were no longer counting rows deleted by the prune but rows inserted into the cache stream. This caused a tight loop.

reivilibre self-requested a review February 26, 2026 17:03

reivilibre requested changes Feb 27, 2026

View reviewed changes

erikjohnston and others added 12 commits April 1, 2026 14:53

Update synapse/storage/databases/main/devices.py

50fd385

Co-authored-by: Olivier 'reivilibre' <oliverw@element.io>

Update docstring to mention what None means

9069a83

Finish docstring for _prune_device_lists_changes_in_room

0179444

Merge remote-tracking branch 'origin/develop' into erikj/remove_old_d…

cfae5da

…evices_in_rooms

Ratelimit how often we delete stuff from the device lists table

de620fa

Add a lower limit to deletion

76e050b

Apply suggestions from code review

19c26bb

Co-authored-by: Olivier 'reivilibre' <olivier@librepush.net>

Note why inserted_Ts can be none

484ca3a

Comment on to_key=None meaning

d41552a

Move delta to correct folder

1bdd726

Fix CRLF

b2e3213

Fix SQLite support

31d2d9e

erikjohnston requested a review from reivilibre April 1, 2026 15:54

reivilibre reviewed Apr 9, 2026

View reviewed changes

Comment thread synapse/storage/databases/main/devices.py Outdated

reivilibre requested changes Apr 9, 2026

View reviewed changes

erikjohnston and others added 4 commits April 10, 2026 11:37

Update synapse/storage/databases/main/devices.py

a1b03f1

Co-authored-by: Olivier 'reivilibre' <oliverw@element.io>

Update synapse/storage/databases/main/devices.py

76eb337

Co-authored-by: Olivier 'reivilibre' <oliverw@element.io>

Fix race checking minimum stream

4712139

Instead of checking it in a separate transaction, check in the transaction we're reading the table from.

Add where clause to inserted_ts index

846491f

erikjohnston force-pushed the erikj/remove_old_devices_in_rooms branch from 5e5c3f2 to fee4c3f Compare April 14, 2026 14:32

erikjohnston requested a review from reivilibre April 14, 2026 15:01

reivilibre requested changes Apr 14, 2026

View reviewed changes

erikjohnston added 3 commits April 14, 2026 17:31

Fix up comments on table

aa2544b

We can assume no rows have yet been deleted

22ac839

Fixup insert

a4dc708

erikjohnston requested a review from reivilibre April 15, 2026 10:46

reivilibre approved these changes Apr 16, 2026

View reviewed changes

erikjohnston added 3 commits April 16, 2026 12:35

Move delete sql

2ef41a9

Add why we need a separate table

e49192d

Lint

eafd1ee

erikjohnston merged commit 2a82859 into develop Apr 17, 2026
78 of 81 checks passed

erikjohnston deleted the erikj/remove_old_devices_in_rooms branch April 17, 2026 10:54

erikjohnston mentioned this pull request Apr 20, 2026

Fix race in new pruning of device lists tables. #19709

Merged

erikjohnston mentioned this pull request Jun 2, 2026

Introduce support for MSC4429: Profile Updates for Legacy Sync #19556

Merged

3 tasks

reivilibre mentioned this pull request Jul 10, 2026

Fix a bug causing device list pruning to skip some rows when the transaction gets retried. #19947

Merged

MadLittleMods added the A-Device-List-Tracking label Jul 13, 2026

	Old `_get_min_device_lists_changes_in_room`	New `_get_max_pruned_device_lists_changes_in_room_txn`
Seed (empty table)	returns 0	returns 0
New rows added with stream_ids 2..=15 (empty table)	returns 2	still returns 0
Prune with prune_before_stream_id 10	now returns MIN(stream_id) = 10	now returns 10 (explicitly stored)

	Old `_get_min_device_lists_changes_in_room`	New `_get_max_pruned_device_lists_changes_in_room_txn`
Seed (populated table with stream_ids 2..=9)	returns MIN(stream_id) = 2	returns MIN(stream_id) - 1 = 1
New rows added with stream_ids 10..=15 (populated table)	still returns 2	still returns 1
Prune with prune_before_stream_id 10	now returns MIN(stream_id) = 10	now returns 10 (explicitly stored)

Uh oh!

Conversation

erikjohnston commented Feb 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

reivilibre left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

reivilibre Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

erikjohnston Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

reivilibre left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

reivilibre Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

erikjohnston Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

erikjohnston commented Apr 14, 2026

Uh oh!

reivilibre left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

reivilibre Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

erikjohnston Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

reivilibre left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

reivilibre Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

erikjohnston Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

erikjohnston commented Feb 17, 2026 •

edited

Loading

erikjohnston Apr 1, 2026 •

edited

Loading

reivilibre Apr 14, 2026 •

edited

Loading