Fix quarantined_media stream not being replicated - #20085
Merged
Conversation
The registration of `QuarantinedMediaStream` in `ReplicationCommandHandler._streams_to_replicate` was missed when the stream was added, so an instance configured as the quarantined_media_changes stream writer never sent RDATA/POSITION for it unless it was the main process. Also add the stream to the `instance_map` config validation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
erikjohnston
force-pushed
the
erikj/quarantined-media-worker-writer-fix
branch
from
August 10, 2026 12:47
55deb63 to
709860a
Compare
erikjohnston
marked this pull request as ready for review
August 10, 2026 12:54
reivilibre
approved these changes
Aug 10, 2026
|
|
||
| continue | ||
|
|
||
| if isinstance(stream, QuarantinedMediaStream): |
Contributor
There was a problem hiding this comment.
does this need docs/development/synapse_architecture/streams.md#cheatsheet-for-creating-a-new-stream to be updated?
Comment on lines
+575
to
+577
| must be routed directly to one of the workers configured as stream writer for the | ||
| `quarantined_media_changes` stream (which must also be able to run the media | ||
| repository, as these endpoints are only registered on media-capable workers): |
Contributor
There was a problem hiding this comment.
The previous language matched what we did for other streams.
But this does seem more clear. Just noting that we should align all of the docs here.
| - will need an [ID generator](https://github.com/element-hq/synapse/blob/4367fb2d078c52959aeca0fe6874539c53e8360d/synapse/storage/databases/main/thread_subscriptions.py#L75) | ||
| - may need [writer configuration](https://github.com/element-hq/synapse/blob/4367fb2d078c52959aeca0fe6874539c53e8360d/synapse/config/workers.py#L177), if there isn't already an obvious source of configuration for which workers should be designated as writers to your new stream. | ||
| - if adding new writer configuration, add Docker-worker configuration, which lets us configure the writer worker in Complement tests: [[1]](https://github.com/element-hq/synapse/blob/4367fb2d078c52959aeca0fe6874539c53e8360d/docker/configure_workers_and_start.py#L331), [[2]](https://github.com/element-hq/synapse/blob/4367fb2d078c52959aeca0fe6874539c53e8360d/docker/configure_workers_and_start.py#L440) | ||
| - Ensure that it's been correctly added to `synapse/replication/tcp/handler.py` and it's `streams_to_replicate` attribute to ensure that changes are actually replicated. |
Contributor
There was a problem hiding this comment.
We already have a point below about "consider whether it may make sense to introduce a handler" which seems like the relevant point here but just needs to be updated to link to what it's talking about and what considerations
Merged
3 tasks
| }, | ||
| ) | ||
|
|
||
| # Quarantining only records a change for media that exists. |
Contributor
There was a problem hiding this comment.
This comment seems irrelevant.
We're not testing existing for non-existing media. And media_id1 doesn't correspond to existing media as far as I can tell
Contributor
There was a problem hiding this comment.
Ohh, this is trying to say that we need to create some actual local media to quarintine.
Suggested change
| # Quarantining only records a change for media that exists. | |
| # Create some media that we can quarintine |
| - will need an [ID generator](https://github.com/element-hq/synapse/blob/4367fb2d078c52959aeca0fe6874539c53e8360d/synapse/storage/databases/main/thread_subscriptions.py#L75) | ||
| - may need [writer configuration](https://github.com/element-hq/synapse/blob/4367fb2d078c52959aeca0fe6874539c53e8360d/synapse/config/workers.py#L177), if there isn't already an obvious source of configuration for which workers should be designated as writers to your new stream. | ||
| - if adding new writer configuration, add Docker-worker configuration, which lets us configure the writer worker in Complement tests: [[1]](https://github.com/element-hq/synapse/blob/4367fb2d078c52959aeca0fe6874539c53e8360d/docker/configure_workers_and_start.py#L331), [[2]](https://github.com/element-hq/synapse/blob/4367fb2d078c52959aeca0fe6874539c53e8360d/docker/configure_workers_and_start.py#L440) | ||
| - Ensure that it's been correctly added to `synapse/replication/tcp/handler.py` and it's `streams_to_replicate` attribute to ensure that changes are actually replicated. |
Contributor
There was a problem hiding this comment.
We could also link to the test added here as an example to sanity check things
MadLittleMods
added a commit
that referenced
this pull request
Aug 11, 2026
…with more references (#20086) The [old link](https://github.com/element-hq/synapse/blob/4367fb2d078c52959aeca0fe6874539c53e8360d/synapse/config/workers.py#L177) only references `thread_subscriptions` once but in the latest state of the code with the [new link](https://github.com/element-hq/synapse/blob/62a4bc46203880dd5034483b0e84156d03a3a8c6/synapse/config/workers.py#L184-L187) there are 5 places in the file to look at. Spawning from seeing a few more changes in #20085 that were missed in #19558 and wondering why I didn't notice before. In fact, some of this clean-up for `thread_subscriptions` (the reference stream from the cheatsheet) wasn't updated until recently (this week) as part of #19556
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The registration of
QuarantinedMediaStreaminReplicationCommandHandler._streams_to_replicatewas missed when the stream was added, so an instance configured as the quarantined_media_changes stream writer never sent RDATA/POSITION for it unless it was the main process.Also add the stream to the
instance_mapconfig validation.Stream was introduced in #19558
Fixes #20080