feat(qwp): stop resending the full symbol dictionary on every message#66
Open
glasstiger wants to merge 1 commit into
Open
feat(qwp): stop resending the full symbol dictionary on every message#66glasstiger wants to merge 1 commit into
glasstiger wants to merge 1 commit into
Conversation
Previously every QWP ingress message re-sent the entire symbol
dictionary, so a connection with many distinct symbols paid to
retransmit the whole dictionary on every message. The client now
sends each symbol id to the server only once per connection.
Memory mode:
- The producer keeps a monotonic "sent" watermark and each frame
carries only the ids above it (a delta section), instead of the
full dictionary from id 0.
- On reconnect or failover the fresh server has an empty dictionary,
so the I/O thread replays the whole dictionary as a catch-up frame
before any post-reconnect traffic, keeping the producer's monotonic
baseline valid across the wire boundary.
Store-and-forward (file mode):
- Each slot persists its dictionary to a dot-prefixed side-file
(PersistedSymbolDict) using write-ahead ordering: new symbols are
appended before the referencing frame is published, so a recovered
or orphan-drained slot on a fresh process can always rebuild the
dictionary that a delta frame references.
- The persistence does not fsync, matching the rest of
store-and-forward, which is process-crash durable (the page cache
survives) but not host-crash durable. A host crash that tears the
dictionary is caught at replay by a guard that fails the send
cleanly ("resend required") instead of transmitting a gapped frame
that would corrupt the table.
Catch-up split:
- The reconnect/recovery catch-up splits across as many frames as the
server's advertised batch cap requires, so a dictionary larger than
the cap is re-registered without any single frame exceeding it. The
frames carry contiguous id ranges and reassemble on the server
exactly as the original per-frame deltas would.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
[PR Coverage check]😍 pass : 297 / 347 (85.59%) file detail
|
glasstiger
added a commit
to questdb/questdb
that referenced
this pull request
Jul 9, 2026
Update the java-questdb-client submodule to de86197, which makes the QWP client register each symbol id with the server only once per connection (delta symbol dictionary) instead of re-sending the whole dictionary on every ingress message. The OSS server already parses delta symbol-dictionary frames, so this is the OSS half of a tandem pair with the client PR questdb/java-questdb-client#66 and needs no server change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Tandem OSS PR (submodule bump): questdb/questdb#7374 — merge together. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every QWP ingress message used to carry the entire symbol dictionary, so a connection that ingests many distinct symbols re-transmitted the whole dictionary on every message. This change makes the client register each symbol id with the server only once per connection and send only new ids (a delta) thereafter, re-registering the full dictionary when a connection is replaced.
The bandwidth saving grows with symbol cardinality and message count; for low-cardinality or short-lived connections it is negligible, and the change adds the costs described under Tradeoffs.
What changed
Memory mode
Store-and-forward (file mode)
PersistedSymbolDict) so a recovered or orphan-drained slot on a fresh process -- which has no in-memory dictionary -- can rebuild what its (non-self-sufficient) delta frames reference.Catch-up split
Durability
The persisted dictionary intentionally does not fsync, matching the rest of store-and-forward: it is process-crash durable (the OS page cache survives a JVM crash) but not host-crash durable. Rather than fsync only the dictionary -- which would not make the frame data itself host-crash durable -- a host crash that tears the dictionary is caught at replay: the send loop detects a delta frame whose start id exceeds the recovered dictionary and fails cleanly with a "resend required" error instead of transmitting a gapped frame that would corrupt the table.
Tradeoffs
Test plan
DeltaDictCatchUpTest-- reconnect catch-up rebuilds the dictionary (memory mode); a large dictionary splits across multiple catch-up frames under a small advertised batch cap and reassembles gap-free.DeltaDictRecoveryTest-- a recovered file-mode slot replays its delta frames against a fresh server; a torn (host-crash) dictionary fails cleanly instead of corrupting.PersistedSymbolDictTest-- side-file append/read/orphan-removal round trips.SelfSufficientFramesTest,ReconnectTest-- full-dict fallback and reconnect replay still hold.SqlFailoverQwpClientLosslessTest(file-mode failover) passes end-to-end against a real server.🤖 Generated with Claude Code