Queue splitting design plan - #4276
Closed
thallgren wants to merge 6 commits into
Closed
Conversation
Base automatically changed from
thallgren/client-rbac-minimization
to
release/v2
August 17, 2026 19:02
A design for personal, header-filtered intercepts of message-queue consumption: a manager-owned queue-agent takes over a declared Kafka topic or RabbitMQ queue and routes messages into per-developer and application shadow queues. Kafka and RabbitMQ are co-equal v1 backends behind a provider-neutral engine contract; implementation is gated on real-broker feasibility prototypes (phase 0). Signed-off-by: Thomas Hallgren <thomas@tada.se>
Both provider gates passed against real brokers (Kafka 4.3.1, RabbitMQ 4.3.4). The one design change: quorum-queue source ownership is monitored through the management API rather than broker-enforced, because quorum queues do not honor the exclusive-consume flag. The proven fencing, handoff, and drain behaviors are folded into the provider engine sections. Signed-off-by: Thomas Hallgren <thomas@tada.se>
The telepresence.io/queue-config annotation gets a strict schema with two-stage validation in pkg/queueconfig: provider-block rules at parse time and pod-template rules (literal source env vars, restricted env sources, TLS secrets already referenced by the container) against the workload spec. pkg/queuestate holds the durable per-workload state document, the activation/deactivation phase machine with per-phase recovery actions, filter overlap detection, and bounded deterministic naming for every broker and Kubernetes resource a split creates. Signed-off-by: Thomas Hallgren <thomas@tada.se>
The Engine interface carries the split lifecycle -- prepare, fenced start, route reconciliation, route drain, stop, application drain, handoff commit, cleanup, and restart recovery -- with the shared at-least-once delivery contract stated once in the package. A provider-agnostic conformance suite encodes the lifecycle scenarios against a per-provider broker probe, and both v1 engines pass it against real brokers: Kafka (franz-go; offset-seeded activation, static-membership and producer-epoch fencing, partition-preserving pump, drain-group residue moves, frontier handback via group-offset commit) and RabbitMQ (amqp091-go; management-API preflight with queue-type parity, single-connection lock-queue fencing, confirmed mandatory publishes, quiesce-and-recheck drains). Signed-off-by: Thomas Hallgren <thomas@tada.se>
Signed-off-by: Thomas Hallgren <thomas@tada.se>
Move the route-filter predicate and the started/stopped/aborted lifecycle (with its cleanup gate) into the engine package, drop DrainApplication's unused Handoff parameter, dedup the RabbitMQ pump-resume and stop-consumer sequences, batch Kafka cleanup's topic/group deletes and group describes, build classification header maps lazily, and replace the Kafka conformance probe's context-keyed scenario tracking with an explicit harness hook. Signed-off-by: Thomas Hallgren <thomas@tada.se>
thallgren
force-pushed
the
thallgren/queue-splitting
branch
from
August 18, 2026 13:08
ed139cf to
1a2e5ba
Compare
Member
Author
|
Superseded by the simplified queue-splitting plan. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Design plan for queue splitting: personal, header-filtered intercepts of message-queue consumption. A manager-owned queue-agent takes over a declared source and routes messages into per-developer and application shadow queues, with Kafka and RabbitMQ as co-equal v1 backends behind a provider-neutral engine contract.
This PR contains only the plan (
docs/plans/queue-splitting/plan.md) for design review; no implementation. Implementation is gated on the phase-0 real-broker feasibility prototypes described in the plan's delivery section.