Add an API to list changes to quarantine state of media - #19558
Merged
Conversation
3 tasks
Sticky events was used as reference for creating this.
turt2live
force-pushed
the
travis/list-quarantined-media-mk2
branch
from
March 16, 2026 19:27
e76dc83 to
6328e78
Compare
turt2live
force-pushed
the
travis/list-quarantined-media-mk2
branch
from
March 16, 2026 19:32
258768e to
f1a35fa
Compare
turt2live
added a commit
that referenced
this pull request
Mar 18, 2026
Just something I noticed while working on #19558 We start the function by setting `total_media_quarantined` to zero, then we do work on the `media_ids`, add the number affected, zero it out (**bug**), do work on `hashes`, add the number of affected rows, then return `total_media_quarantined`. ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [x] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
Contributor
|
Thanks for pushing through all of the hard stuff @turt2live 🦏 |
3 tasks
turt2live
added a commit
that referenced
this pull request
Apr 9, 2026
…#19677) Following up on #19558 (comment) Changelog for this PR is intended to overlap with the above PR. `get_current_quarantined_media_stream_id` wasn't being used anywhere else, so we can replace it like we do in this PR. ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [x] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: Eric Eastwood <erice@element.io> Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>
MadLittleMods
referenced
this pull request
Apr 10, 2026
3 tasks
reivilibre
pushed a commit
that referenced
this pull request
Apr 16, 2026
Merged
3 tasks
MadLittleMods
added a commit
that referenced
this pull request
May 14, 2026
…th unbounded token (#19644) Spawning from trying to find the proper way to wait for a token, see #19558 (comment) - Update `wait_for_stream_token(...)` patterns so validation/sanitization is handled upstream in usage. - Fix sync waiting for bounded token but using unbounded token to fetch data. Noticed while working on adding the new method. Part of #19647
MadLittleMods
added a commit
that referenced
this pull request
May 15, 2026
…it_for_stream_token(...)` (#19764) In order to be able to use `wait_for_stream_token(...)`, we have to add the `quarantined_media` stream to the `StreamToken`. Even though we don't care about `/sync`'ing `quarantined_media`, this aligns with the future where all endpoints should probably use `StreamToken`, see #19647 Follow-up to #19558 and #19644
FrenchGithubUser
pushed a commit
to famedly/synapse-upstreaming
that referenced
this pull request
Jun 12, 2026
Just something I noticed while working on element-hq#19558 We start the function by setting `total_media_quarantined` to zero, then we do work on the `media_ids`, add the number affected, zero it out (**bug**), do work on `hashes`, add the number of affected rows, then return `total_media_quarantined`. ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [x] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
FrenchGithubUser
pushed a commit
to famedly/synapse-upstreaming
that referenced
this pull request
Jun 12, 2026
…portdb` (element-hq#19675) Part of element-hq#19671 Spawning from [discussion in `#synapse-dev:matrix.org`](https://matrix.to/#/!i5D5LLct_DYG-4hQprLzrxdbZ580U9UB6AEgFnk6rZQ/$Z3nqbH0Qy21FWC3qJOim6LSRCRpJ3pxV5DLXm98IA6I?via=element.io&via=matrix.org&via=beeper.com) with roots in element-hq#19558 (comment). As trialed/discovered by @turt2live alongside @reivilibre and @clokep ❤️ ### Why is this necessary? If you forget to add `_setup_sequence(...)`, you can run into the following error if there is 1 row in SQLite and then you use the `portdb` script to try to migrate to Postgres (as [explained](https://matrix.to/#/!i5D5LLct_DYG-4hQprLzrxdbZ580U9UB6AEgFnk6rZQ/$mHU6dcTNL7NMfKBCJUekCh7vDj1lr1GDjriZQl7oeeU?via=element.io&via=matrix.org&via=beeper.com) by @reivilibre) ``` Postgres sequence 'quarantined_media_id_seq' is inconsistent with associated stream position of 'quarantined_media' in the 'stream_positions' table. ```
FrenchGithubUser
pushed a commit
to famedly/synapse-upstreaming
that referenced
this pull request
Jun 12, 2026
…9558) Fixes element-hq#19352 (See issue for history of this feature and previous PRs) > First, a [naive implementation](element-hq#19268) of the endpoint was introduced, but it quickly ran into [performance issues on query](element-hq#19312) and [long startup times](element-hq#19349), leading to its [removal](element-hq#19351). It also didn't actually work, and would fail to expose media when it was "unquarantined", so a [partial fix](element-hq#19308) was attempted, where the suggested direction is to use a [stream](https://element-hq.github.io/synapse/latest/development/synapse_architecture/streams.html#cheatsheet-for-creating-a-new-stream) instead of a timestamp column. This PR re-introduces the API building on the previous feedback: * Adds a stream which tracks when media becomes (un)quarantined. * Runs a background update to capture already-quarantined media. * Adds a new admin API to return rows from the stream table. We track both quarantine and unquarantine actions in the stream to allow downstream consumers to process the records appropriately. Namely, to allow our Synapse exchange in HMA to remove hashes for unquarantined media (use case further explained in the [issue](element-hq#19352)). **Note**: This knowingly does not capture all cases of media being quarantined. Other call sites are lower priority for T&S, and can be addressed in a future PR. ~~An issue will be created after this PR is merged to track those sites.~~ element-hq#19672 ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [x] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: turt2live <1190097+turt2live@users.noreply.github.com> Co-authored-by: Eric Eastwood <madlittlemods@gmail.com> Co-authored-by: Eric Eastwood <erice@element.io>
FrenchGithubUser
pushed a commit
to famedly/synapse-upstreaming
that referenced
this pull request
Jun 12, 2026
…element-hq#19677) Following up on element-hq#19558 (comment) Changelog for this PR is intended to overlap with the above PR. `get_current_quarantined_media_stream_id` wasn't being used anywhere else, so we can replace it like we do in this PR. ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [x] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: Eric Eastwood <erice@element.io> Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>
FrenchGithubUser
pushed a commit
to famedly/synapse-upstreaming
that referenced
this pull request
Jun 12, 2026
…q#19694) Fixes element-hq#19692 Introduced by element-hq#19558 --------- Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>
FrenchGithubUser
pushed a commit
to famedly/synapse-upstreaming
that referenced
this pull request
Jun 12, 2026
…th unbounded token (element-hq#19644) Spawning from trying to find the proper way to wait for a token, see element-hq#19558 (comment) - Update `wait_for_stream_token(...)` patterns so validation/sanitization is handled upstream in usage. - Fix sync waiting for bounded token but using unbounded token to fetch data. Noticed while working on adding the new method. Part of element-hq#19647
FrenchGithubUser
pushed a commit
to famedly/synapse-upstreaming
that referenced
this pull request
Jun 12, 2026
…it_for_stream_token(...)` (element-hq#19764) In order to be able to use `wait_for_stream_token(...)`, we have to add the `quarantined_media` stream to the `StreamToken`. Even though we don't care about `/sync`'ing `quarantined_media`, this aligns with the future where all endpoints should probably use `StreamToken`, see element-hq#19647 Follow-up to element-hq#19558 and element-hq#19644
erikjohnston
added a commit
that referenced
this pull request
Jul 2, 2026
…te (#19901) The `flag_existing_quarantined_media` background update (added in #19558, shipped in v1.152.0) back-populates the `quarantined_media_changes` table with media that was already quarantined. It has two bugs. ### 1. Some quarantined remote media is silently skipped The remote-media query paged through `remote_media_cache` with: ```sql WHERE quarantined_by IS NOT NULL AND media_origin >= ? AND media_id > ? ``` This ANDs the two key columns independently rather than comparing them as a tuple. Once an origin has been fully processed (e.g. `media_id` reaches `zzz` for origin `a.example`), rows in a *later* origin whose `media_id` is `<=` the last processed `media_id` (e.g. `b.example` / `aaa`) fail the `media_id > ?` test and are never flagged. Fixed by using a proper row-value tuple comparison: ```sql WHERE quarantined_by IS NOT NULL AND (media_origin, media_id) > (?, ?) ``` Both the minimum supported SQLite (3.37.2) and PostgreSQL support row-value comparisons. ### 2. Exhausted queries keep re-running every iteration `flag_quarantined` ran *both* the local and remote queries on every iteration. When one table was exhausted but the other still had rows, the update kept returning a positive count, so the finished table's (now empty) query needlessly re-ran on every subsequent iteration until the whole update completed. This could add significant time to the transaction, as since the rows were deleted it could scan a significant portion of the table each time. Fixed by tracking per-table completion (`local_done` / `remote_done`) in the background-update progress and skipping a table's query once it has returned an empty batch. The progress dict was also restructured into an incremental build for readability.
netbsd-srcmastr
pushed a commit
to NetBSD/pkgsrc
that referenced
this pull request
Jul 7, 2026
Tested on NetBSD 10 amd64 with 2026Q2 environment. # Synapse 1.156.0 (2026-07-07) ## Features - Expose [MSC4354 Sticky Events](matrix-org/matrix-spec-proposals#4354) over [MSC4186 (Simplified) Sliding Sync](matrix-org/matrix-spec-proposals#4186). ([\#19591](element-hq/synapse#19591)) - Stabilize support for sending ephemeral events to application services, as per [MSC2409](matrix-org/matrix-spec-proposals#2409). Contributed by @jason-famedly @ Famedly. ([\#19758](element-hq/synapse#19758)) - Include `allowed_room_ids` in the `/summary` client-server API response for rooms with restricted join rules, as required by Matrix 1.15. Contributed by @FrenchGithubUser @famedly. ([\#19762](element-hq/synapse#19762)) - [MSC4140: Cancellable delayed events](matrix-org/matrix-spec-proposals#4140): Allow authentication on delayed event management endpoints (such as `/restart`) to bypass ratelimits for unauthenticated requests based on the client IP address. ([\#19794](element-hq/synapse#19794)) - Add new metric `synapse_non_deactivated_user_count` which tracks the number of non-deactivated users in the database, split by `app_service`. ([\#19848](element-hq/synapse#19848)) - The `GET /_matrix/client/unstable/org.matrix.msc1763/retention/configuration` endpoint is now provided when retention is enabled and `experimental_features.msc1763_enabled` is enabled, based on [MSC1763](matrix-org/matrix-spec-proposals#1763). ([\#19853](element-hq/synapse#19853)) - Add experimental support for [MSC4491: Invite reasons in room creation](matrix-org/matrix-spec-proposals#4491). ([\#19874](element-hq/synapse#19874)) # Synapse 1.155.0 (2026-06-16) # Synapse 1.154.0 (2026-06-04) ## Features - Add support for [MSC4452: Preview URL capabilities API](matrix-org/matrix-spec-proposals#4452) which exposes a `io.element.msc4452.preview_url` capability. If `experimental_features.msc4452_enabled` is `true`, the `/_matrix/(client/v1/media|media/v3)/preview_url` endpoint now responds with a 403 status code when the capability is disabled. ([\#19715](element-hq/synapse#19715)) # Synapse 1.153.0 (2026-05-19) ## Features - Make ACLs apply to EDUs per [MSC4163](matrix-org/matrix-spec-proposals#4163). ([\#18475](element-hq/synapse#18475)) - Stabilize [MSC3266: Room summary API](matrix-org/matrix-spec-proposals#3266), removing the experimental config flag `msc3266_enabled`. Contributed by @dasha-uwu. ([\#19720](element-hq/synapse#19720)) - Partial [MSC4311](matrix-org/matrix-spec-proposals#4311) implementation: `m.room.create` is now a required part of stripped `invite_state`/`knock_state` . Contributed by @FrenchGithubUser @famedly. ([\#19722](element-hq/synapse#19722)) - Expose `tombstoned` and `replacement_room` in room details on admin API endpoint `GET /_synapse/admin/v1/rooms/<room_id>`. Contributed by Noah Markert. ([\#19737](element-hq/synapse#19737)) # Synapse 1.152.1 (2026-05-07) # Synapse 1.152.0 (2026-04-28) ## Features - Add a ["Listing quarantined media changes" Admin API](https://element-hq.github.io/synapse/latest/admin_api/media_admin_api.html#listing-quarantined-media-changes) for retrieving a paginated record of when media became (un)quarantined. ([\#19558](element-hq/synapse#19558), [\#19677](element-hq/synapse#19677), [\#19694](element-hq/synapse#19694)) - Advertise [MSC4445](matrix-org/matrix-spec-proposals#4445) sync timeline order in `unstable_features`. ([\#19642](element-hq/synapse#19642)) - Report the Rust compiler version used in the Prometheus metrics. Contributed by Noah Markert. ([\#19643](element-hq/synapse#19643)) - Passthrough 'article' and 'profile' OpenGraph metadata on URL preview requests. ([\#19659](element-hq/synapse#19659)) - Add a way to re-sign local events with a new signing key. ([\#19668](element-hq/synapse#19668)) - Support [MSC4450: Identity Provider selection for User-Interactive Authentication with Legacy Single Sign-On](matrix-org/matrix-spec-proposals#4450). ([\#19693](element-hq/synapse#19693)) - Add experimental support for [MSC4242](matrix-org/matrix-spec-proposals#4242): State DAGs. Excludes federation support. ([\#19424](element-hq/synapse#19424)) - Adds [Admin API](https://element-hq.github.io/synapse/latest/usage/administration/admin_api/index.html) endpoints to list, fetch and delete user reports. ([\#19657](element-hq/synapse#19657)) - Reduce database disk space usage by pruning old rows from `device_lists_changes_in_room`. ([\#19473](element-hq/synapse#19473), [\#19709](element-hq/synapse#19709)) # Synapse 1.151.0 (2026-04-07) ## Features - Add stable support for [MSC4284](matrix-org/matrix-spec-proposals#4284) Policy Servers. ([\#19503](element-hq/synapse#19503)) - Update and stabilize support for [MSC2666](matrix-org/matrix-spec-proposals#2666): Get rooms in common with another user. Contributed by @tulir @ Beeper. ([\#19511](element-hq/synapse#19511)) - Updated experimental support for [MSC4388: Secure out-of-band channel for sign in with QR](matrix-org/matrix-spec-proposals#4388). ([\#19573](element-hq/synapse#19573)) - Stabilize `room_version` and `encryption` fields in the space/room `/hierarchy` API (part of [MSC3266](matrix-org/matrix-spec-proposals#3266)). ([\#19576](element-hq/synapse#19576)) - Introduce a [configuration option](https://element-hq.github.io/synapse/latest/usage/configuration/config_documentation.html#matrix_authentication_service) to allow using HTTP/2 over plaintext when Synapse connects to Matrix Authentication Service. ([\#19586](element-hq/synapse#19586)) ## Deprecations and Removals - Remove support for [MSC3852: Expose user agent information on Device](matrix-org/matrix-spec-proposals#3852) as the MSC was closed. ([\#19430](element-hq/synapse#19430)) # Synapse 1.150.0 (2026-03-24) ## Features - Add experimental support for the [MSC4370](matrix-org/matrix-spec-proposals#4370) Federation API `GET /extremities` endpoint. ([\#19314](element-hq/synapse#19314)) - [MSC4140: Cancellable delayed events](matrix-org/matrix-spec-proposals#4140): When persisting a delayed event to the timeline, include its `delay_id` in the event's `unsigned` section in `/sync` responses to the event sender. ([\#19479](element-hq/synapse#19479)) - Expose [MSC4354 Sticky Events](matrix-org/matrix-spec-proposals#4354) over the legacy (v3) /sync API. ([\#19487](element-hq/synapse#19487)) - When Matrix Authentication Service (MAS) integration is enabled, allow MAS to set the user locked status in Synapse. ([\#19554](element-hq/synapse#19554))
This was referenced Aug 10, 2026
erikjohnston
added a commit
that referenced
this pull request
Aug 10, 2026
The registration of `QuarantinedMediaStream` in `ReplicationCommandHandler._streams_to_replicate` was missed when the stream was added, so an instance configured as the quarantined_media_changes stream writer never sent RDATA/POSITION for it unless it was the main process. Also add the stream to the `instance_map` config validation. Stream was introduced in #19558 Fixes #20080 --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Merged
3 tasks
MadLittleMods
added a commit
that referenced
this pull request
Aug 11, 2026
…with more references (#20086) The [old link](https://github.com/element-hq/synapse/blob/4367fb2d078c52959aeca0fe6874539c53e8360d/synapse/config/workers.py#L177) only references `thread_subscriptions` once but in the latest state of the code with the [new link](https://github.com/element-hq/synapse/blob/62a4bc46203880dd5034483b0e84156d03a3a8c6/synapse/config/workers.py#L184-L187) there are 5 places in the file to look at. Spawning from seeing a few more changes in #20085 that were missed in #19558 and wondering why I didn't notice before. In fact, some of this clean-up for `thread_subscriptions` (the reference stream from the cheatsheet) wasn't updated until recently (this week) as part of #19556
FrenchGithubUser
pushed a commit
to famedly/synapse
that referenced
this pull request
Aug 13, 2026
The registration of `QuarantinedMediaStream` in `ReplicationCommandHandler._streams_to_replicate` was missed when the stream was added, so an instance configured as the quarantined_media_changes stream writer never sent RDATA/POSITION for it unless it was the main process. Also add the stream to the `instance_map` config validation. Stream was introduced in element-hq/synapse#19558 Fixes element-hq/synapse#20080 --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #19352
(See issue for history of this feature and previous PRs)
This PR re-introduces the API building on the previous feedback:
We track both quarantine and unquarantine actions in the stream to allow downstream consumers to process the records appropriately. Namely, to allow our Synapse exchange in HMA to remove hashes for unquarantined media (use case further explained in the issue).
Note: This knowingly does not capture all cases of media being quarantined. Other call sites are lower priority for T&S, and can be addressed in a future PR.
An issue will be created after this PR is merged to track those sites.#19672Pull Request Checklist
EventStoretoEventWorkerStore.".code blocks.