docs: introduce Connect section, connect-string reference, QWP stubs#444
Open
bluestreak01 wants to merge 147 commits into
Open
docs: introduce Connect section, connect-string reference, QWP stubs#444bluestreak01 wants to merge 147 commits into
bluestreak01 wants to merge 147 commits into
Conversation
Consolidate how-applications-talk-to-QuestDB content under a single Connect supersection. Rewrite the ingestion overview as QWP-native "Connect to QuestDB", add a comprehensive connect-string reference at documentation/client-configuration/, and scaffold the Wire Protocols sub-section under documentation/protocols/. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
🤖 Component Converter ReminderA component in We are creating markdown correspondents of every path (e.g. questdb.com/docs/quick-start/ → questdb.com/docs/quick-start.md) for LLM consumption. Quick Check
💡 This is a friendly reminder, not a blocker. Ignore if not applicable. |
|
🚀 Build success! Latest successful preview: https://preview-444--questdb-documentation.netlify.app/docs/ Commit SHA: 4baccbc
|
UDP documentation is deferred. Removes the stub page, sidebar entry, and links from ingress, egress, overview, and connect-string pages.
Closed
- Add "Why implement a QWP client" pitch and "Client lifecycle" narrative so new implementers can orient before diving into encoding details. - Spell out sequence numbering (server-assigned by receive order, not in the wire header), Gorilla first-DoD anchor, decimal scale formula (value = unscaled / 10^scale), and VARCHAR offset endianness — closes silent-wrong-guess risks for one-shot client generation. - Collapse Symbol section to WebSocket-only (per-table dict is UDP) and drop the now-stranded per-table example. - Document the practical WebSocket frame cap: http.recv.buffer.size (default 2 MiB) is the real ceiling, not the 16 MB protocol limit; exceeding it returns close code 1009 MESSAGE_TOO_BIG. - Fill out durable-ack semantics: watermark trails OK, empty messages trivially durable, reconnects discard in-flight tracking. - Note X-QWP-Client-Id may influence version selection. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…/questdb/documentation into docs/connect-and-qwp-scaffolding # Conflicts: # documentation/protocols/qwp-ingress-websocket.md
- Add "Why implement a QWP query client" pitch and "Client lifecycle" narrative paralleling the ingress doc; surfaces the java-questdb-client reference impl link upfront. - Document the practical WebSocket frame cap on /read/v1: client-to-server frames (QUERY_REQUEST in particular) are bounded by http.recv.buffer.size (default 2 MiB), not the 16 MiB protocol limit; oversized frames are rejected with close code 1009 MESSAGE_TOO_BIG. - Clarify X-QWP-Max-Batch-Rows only asks for smaller batches than the server default (clamps to server's hard limit). - Tighten NULL sentinel docs: FLOAT/DOUBLE sentinel is *any* NaN (incl. 0.0/0.0); IPv4 0.0.0.0 and all-ones GEOHASH cannot round-trip as non-null. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…sage Brokers up to Connect - Add a "Use QWP for new clients" tip callout to ilp/overview.md naming QWP's wins (binary, type-rich, faster, failover, store-and-forward) and framing ILP as the path for InfluxDB / Telegraf / Kafka / Flink users who already emit ILP. - Shorter callout on ilp/columnset-types.md reframing the page as "extensions on top of the InfluxDB type model" and noting QWP exposes the full QuestDB type system natively (no suffix encoding, no casts). - Operator-facing callout on ilp/advanced-settings.md flagging this page as the legacy ILP tuning surface and pointing new deployments at QWP. - Sidebar: lift Java Embedded and Message Brokers out from under ILP (they're protocol-agnostic delivery mechanisms, not ILP sub-pages). Final Connect order: Overview, Connect string, Date to Timestamp, Client Libraries, Message Brokers, Compatibility Protocols, Java Embedded, Wire Protocols. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…eader) The egress endpoint /read/v1 is asymmetric on the wire: server-to-client frames carry the 12-byte QWP header, but client-to-server frames start directly with msg_kind — no QWP header. Including the header makes the server read 0x51 (the ASCII 'Q' of "QWP1") as an unknown msg_kind and close the WebSocket with code 1006, partway through send. Verified against server (QwpEgressUpgradeProcessor.dispatchEgressMessage calls peekMsgKind at offset 0 of the WS frame body) and Java reference client (QwpEgressIoThread.sendQueryRequest writes msg_kind as the first byte, no header). The upstream wire-egress.md spec is wrong on this point and should be filed separately. - Rewrite Message structure section with two ASCII diagrams (server-to- client with header, client-to-server without) and a warning callout naming the symptom and the reason (server keeps the header for RESULT_BATCH's flags + payload_length; client control frames have no analogous need). - Fix Example 1: drop the bogus 12-byte header from the QUERY_REQUEST hex dump; RESULT_BATCH / RESULT_END below unchanged. - Client lifecycle step 4: inline note that the binary frame body starts directly with msg_kind for client-to-server frames. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…/questdb/documentation into docs/connect-and-qwp-scaffolding
New page (documentation/connect/agents.md) covering how AI agents operate QuestDB across three angles: protocols, tooling, and operational practices. Positioning: - QWP egress is the recommended path for SQL execution (DDL + streaming SELECT), with native client libraries when available and the protocol spec for clean-room implementations. - QWP ingress is the recommended path for all writes (bulk and sustained), including local-file uploads — the recipe explicitly calls out the failure mode where agents reach for read_parquet/read_csv/COPY, which require server-side filesystem access. - REST is positioned for schema discovery and small ad-hoc queries that fit in a single HTTP response. - PGWire and /imp are intentionally not recommended (superseded by QWP). - No MCP framing: an MCP server would just wrap REST + QWP without adding capability, so the page tells agents to use the underlying protocols directly. Includes a Recipes section seeded with the local-file upload recipe; links to the existing Getting Started > AI Coding Agents page for the tooling quickstart and the QuestDB / TSBS Claude skills. Sidebar: - Add Connect > Agents (between Client Libraries and Message Brokers). - Move Date to Timestamp inside Client Libraries (cross-cutting reference for all language clients). - Move Connect string inside Client Libraries as the first item (config schema shared by every QWP client). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…/questdb/documentation into docs/connect-and-qwp-scaffolding
…matches The Go client now retries transport outages indefinitely: reconnect_max_duration_millis bounds only the blocking sync initial connect, and the budget-exhausted terminal event kind is gone. Update the reconnect key tables, connection-event kinds, and the ingress-reconnect section accordingly; document the new WithQuestDBDrainerListener facade option and query_close_timeout_ms key. Also fix documented behavior that diverged from the code: multi-address WithAddress usage (single comma-separated string), Exec never returns QwpFailoverReset, single-addr failover retries the same host, GEOHASH has a value accessor, Query streams during iteration rather than blocking, producer backpressure applies in memory mode too, the fixed-width decimal columns are the QwpSender-only ones, and the failover example now genuinely accumulates and clears on reset.
…erminal and retriable rejections
…nnect Reflect net-questdb-client PR #76 in the .NET client docs: - Query egress API moved from the push/handler model (ExecuteAsync + QwpColumnBatchHandler with OnBatch/OnEnd/OnExecDone/ OnError/OnFailoverReset) to a DbDataReader-style pull cursor: ExecuteReaderAsync -> IQwpQueryReader -> while (ReadBatchAsync) reader.Current. Errors now throw QwpQueryException (Status/ServerMessage); DDL/DML report via reader.OpType / reader.RowsAffected; mid-stream failover via reader.FailoverReset; cooperative cancel via reader.Cancel()/WasCancelled. - Pool query surface updated: ExecuteReaderAsync / NewQuery().Sql().Binds() .ExecuteReaderAsync() (dropped ExecuteSqlAsync / .Handler()). - Add "Tolerant startup (lazy_connect)" section documenting the QuestDBClient handle flag: async ingest connect, lazy read-pool first-use connect, and the up-front conflict rejections. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The docs claimed disposing a ws/wss sender flushes and drains in-flight ACKs. Per net-questdb-client PR #76 source, dispose (standalone and pooled) is pure resource release: it does NOT send, does NOT wait for ACKs, and DISCARDS any buffered-but-unsent rows. Callers must Send()/SendAsync() before disposing. Verified from source and corrected: - Dispose never sends; buffered rows are dropped — fixed the quick start, pool quick start (which previously lost data with no Send before dispose), "Borrowing a sender", "Closing the sender", and the delivery-tracking intro. - Send() semantics: a standalone ws sender DRAINS (flush + await ACK, bounded by close_flush_timeout_millis, throws on timeout); a pooled sender flushes to the ring and returns immediately (confirm via Flush / IQuestDBClient.Flush). Fixed "Explicit flushing", backpressure, and the async-error-handling intro. - close_flush_timeout_millis now bounds the standalone Send drain (and is the default Flush timeout), not "the drain at dispose" — fixed the config table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The column-major chunk section of the pooled C/C++ guide used setter names that don't exist in the c-questdb-client headers. The designated-timestamp setter appeared in the runnable C and C++ examples, so both failed to compile. Correct the names to match column_sender.h / column_sender.hpp (verified against v7.0.0): - designated_timestamp_* -> at_nanos / at_micros / at_millis / at_seconds - column_ts_nanos / column_ts_micros -> column_ts (+ column_sender_ts_unit) - column_date_millis -> column_date - column_varchar -> column_str - symbol_dict_i8 / _i16 / _i32 -> symbol_i8 / _i16 / _i32 Also tidy the page for shipping: - drop the internal branch name from the Draft admonition - rename the slug /connect/clients/c-api-reference -> /connect/clients/c-and-cpp-pooled (it's a guide, not a generated ABI reference) and update the Rust cross-link - note the QUESTDB_CLIENT_ENABLE_ARROW build requirement in the C++ Arrow example
Replace the standalone line_sender guide with the pooled guide at /connect/clients/c-and-cpp. Inline the content the pooled page deferred to the old guide (unsupported auth paths, mid-stream failover duplicate handling), drop the draft banner and the extra sidebar entry, and clean up prose (em dashes, meta-commentary). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wrap all paired C/C++ examples in synced tab groups (groupId c-cpp, C++ default), add C examples for the previously C++-only concurrency and production sections, and rework prose that assumed the stacked layout (Connecting laziness, Querying reader error types, parameterised query wrap-up). All standalone examples syntax-checked against the client headers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
column_sender.h includes line_sender.h (and column_sender.hpp includes line_sender.hpp), so both languages need the same two headers: one for writing, one for querying. Verified all 14 examples still compile against the client headers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The quick start's read-back returned zero rows on a fresh database: the flush ack confirms server acceptance, while query visibility follows WAL apply. Pause before the read-back, document ack vs visibility in the durability section, and add an installation section (FetchContent, questdb_client target, C11/C++17, reader and Arrow build options). Both quick-start tabs verified against a nightly server.
The client core is Rust compiled from source, so a C++ developer without rustc hits an opaque Corrosion configure failure. Name the prerequisite, the exact error, and the first-build compile time. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Verified against the client source (jh_conn_pool_refactor): flush cadence numbers, wait timeout 0 semantics, tls_ca default and chooser, multi-endpoint topology assumptions, sender_id slot semantics, pool key tuning rationale, the full bind surface incl. typed NULL binds, and the close_flush_timeout_millis name fix. Also fixes the production example, whose durable wait silently degraded to ok without request_durable_ack=on, and drops the standalone reader (the page documents the pooled API). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…scaffolding # Conflicts: # documentation/connect/clients/date-to-timestamp-conversion.md # documentation/connect/message-brokers/redpanda.md # documentation/query/functions/meta.md
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reorder into get-running / core-model / do-the-work / production / reference tiers. Slim "The pool" to the concept and move its API-at-a-glance table down to a Reference section; relocate Pool keys and Sizing from Connecting into the production tier. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add an "Arrow result batches" subsection covering the egress Arrow C Data Interface export (C++ next_arrow_batch / C reader_cursor_next_arrow_batch and _compact), with ownership, compact-symbol, and schema-drift notes. Generalize the CMake QUESTDB_ENABLE_ARROW bullet to cover both ingestion and result batches. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make the egress Arrow section a peer H2 of "Arrow ingestion" instead of a subsection of Querying, and cross-link the two directions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop the "common insert and query path" claim; the quick start exercises both APIs in one runnable program, but ingestion and query paths are usually segregated in production. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The client pool no longer has a single-active-borrower rule in disk-backed store-and-forward mode: with sf_dir set it mints one kind-scoped slot per borrowed sender (<sender_id>-col-<index> / <sender_id>-row-<index>), pool_size / pool_max keep their normal per-kind meaning, and dirty slots replay automatically at connect. - c-and-cpp.md: rewrite the Pool keys SF paragraphs; drop single-borrower claims from the column-major and concurrency sections; document the at-cap close-wait exception and connect-time recovery; fix the false "sender_id and the other sf_* keys require an explicit sf_dir" claim; replace the 3 * pool_max ceiling with a per-kind budget. - rust.md: state the minted-slot model in the store-and-forward note. - connect-string.md: document sender_id as the slot base for pooled senders. Verified against c-questdb-client jh_conn_pool_refactor (4a04a14) with runnable C++ probes: concurrent row+column borrows with sf_dir, minted slot directories on disk, and sf_* keys accepted in memory mode. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The client (jh_conn_pool_refactor 2692bf9) rejects a durable-level wait on a pool opened without request_durable_ack=on, matching flush_and_wait, instead of silently degrading to the ok level. Also restructure the dense "wait = observation" paragraph into sub-bullets (ack levels, ack vs visibility, no-progress deadline) and name the background runner as the component that talks to the QuestDB server. Verified with C++ probes against a local nightly: both durable-level calls fail up front with invalid_api_call on both sender kinds, ok-level waits unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Lead with a runnable write-and-query quick start, then progressively cover connection, ingestion, querying, delivery, failover, sizing, and lifecycle guidance. Document the source-validated pool and borrowed-handle APIs, feature gates, and operational tradeoffs.
Reorder the Rust guide from quick start through connection, API choice, usage, and operational guidance before its reference sections. Reflect the default Rust query reader and Zstandard support, simplify the Cargo dependency, and remove the obsolete CMake reader toggle from the C/C++ guide.
The c-questdb-client now uses ws/wss exclusively; drop the qwpws/qwpwss alias mention from the C/C++ and Rust client pages. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The progress callback is now telemetry-only; without a reset callback a post-delivery failure returns FailoverWouldDuplicate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Consolidates how-applications-talk-to-QuestDB content under a single Connect supersection. The native QWP clients, the compatibility protocols (ILP / PGWire / REST), and the wire-protocol specifications now share one home. The connect-string reference at
documentation/client-configuration/connect-string.mdanchors the section as the comprehensive knob doc.What's new
Sidebar restructure
Connectsection absorbs Client Libraries, Compatibility Protocols, and Wire Protocols.Connect → Client Libraries.Connect → Compatibility Protocols.Connect → Wire Protocolsfor the client-implementer audience.Connect → Overview (
documentation/ingestion/overview.md)Connect → Connect string reference (
documentation/client-configuration/connect-string.md, new)*Applies to: ingress.*/*Applies to: ingress and egress.*).sf_dirpath handling (no shell expansion, parent must exist).sender_idvalidation (allowed characters, lock-collision semantics).Wire Protocols scaffolding (
documentation/protocols/, new)Touch-ups
#first-party-clients→#client-librariesincapacity-planning.md,monitoring-alerting.md,datatypes/overview.md, andsrc/components/Resources/index.tsx.date-to-timestamp-conversionfrontmatter description from a bare language list to an actual summary.Project handoff (
ONBOARDING.md)Bundle B / A / C team handoff document captured at repo root — assignments, coordination rules, first-prompt templates for the remaining work streams (Wire-protocol content, Client failover, Store-and-forward concepts). Reviewers can opt to keep or remove from this PR.
Out of scope (intentionally)
Connect → Compatibility Protocols → ILP. The connect-string reference is QWP-native; ILP-specific knobs (protocol_version,retry_timeout,request_timeout,request_min_throughput) are not duplicated.Known follow-ups
sf_durability=flush|append, theon_*_errorfamily) so LLM auto-completion is less likely to surface them as available.Add per-language file moves and HTTP-side URL redirects in a separate restructure PR.Redirects now handled in questdb/questdb.io#2917.Test plan
yarn build— 353 pages generated, no broken links / anchorstarget/zoneframing matches the QuestDB Enterprise feature storyONBOARDING.mdshould remain at repo root🤖 Generated with Claude Code