Skip to content

Commit 2872cb4

Browse files
feat(webhooks): verify_and_parse_* API for compressed payloads (CHA-3071) (#230)
* feat(webhooks): add verify_and_decode_webhook for compressed payloads (CHA-3071) Stream Chat backend can now compress outbound webhook payloads with gzip and, for SQS / SNS firehose delivery, base64-wrap the compressed bytes so they remain valid UTF-8 over the queue. Add two new client methods that let customers decompress + verify in a single call: - decompress_webhook_body(body, content_encoding=None, payload_encoding=None) primitive decode that handles gzip and/or base64 - verify_and_decode_webhook(body, x_signature, content_encoding=None, payload_encoding=None) decode + HMAC-SHA256 verify Both are exposed on the sync StreamChat and async StreamChatAsync clients through the shared StreamChatInterface base, mirroring the existing verify_webhook helper. The existing verify_webhook signature and behavior are unchanged for backward compatibility. A new WebhookSignatureError (extends StreamAPIException) is raised on signature mismatch, malformed gzip, or malformed base64. Unsupported encoding values raise ValueError with a message that points at the supported algorithm (gzip). The decoding logic lives in stream_chat/webhook.py so it can be tested without instantiating an HTTP client. The new tests cover the cross-SDK contract: passthrough, gzip round-trip, base64 round-trip, base64 + gzip (SQS / SNS shape), case-insensitive aliases, every unsupported content_encoding (br / brotli / zstd / deflate / compress / lz4), unsupported payload_encoding (hex / url / binary), invalid gzip / base64 input, and three signature-mismatch variants (wrong signature, signature over compressed bytes, signature over wrapped bytes). Docs: webhooks_overview.md gets a "Compressed webhook bodies" section with Django, Flask, and SQS / SNS usage examples. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tests): satisfy isort import-block rule Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(webhooks): switch to verify_and_parse_* API (CHA-3071) Replaces the earlier verify_and_decode_webhook surface with the cross-SDK contract documented at https://getstream.io/chat/docs/node/webhooks_overview/. Module-level helpers in stream_chat.webhook: Primitives: ungzip_payload - gzip magic-byte detection + inflate decode_sqs_payload - base64 then ungzip-if-magic decode_sns_payload - alias for decode_sqs_payload verify_signature - constant-time HMAC-SHA256 comparison parse_event - JSON -> dict (typed event lands later) Composite (return parsed event dict): verify_and_parse_webhook verify_and_parse_sqs verify_and_parse_sns The composite functions auto-detect compression from body bytes, so the same handler stays correct whether or not Stream is currently compressing payloads, and behind middleware that auto-decompresses. Client instance methods (StreamChat / StreamChatAsync) mirror the three composite helpers with api_secret pulled from the client. The legacy verify_webhook(body, x_signature) -> bool boolean helper is unchanged for backward compatibility. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(webhooks): drop redundant binascii.Error in except (B014) binascii.Error is a subclass of ValueError, so listing both in the except clause triggers flake8-bugbear B014. Catching ValueError alone covers both cases. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(webhooks): use 2-byte gzip magic per RFC 1952 (CHA-3071) RFC 1952 defines the gzip magic number as the two-byte sequence 1F 8B; the third byte (CM) is informational and not part of the identifier. Trim the magic check from three bytes to two to match the spec and stay consistent with the reference implementations in the public docs. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(webhooks): make verify_signature robust against malformed signatures Previously, passing a signature with non-ASCII bytes (e.g. b"\xff..."), a non-ASCII unicode string, or a non-string type would raise UnicodeDecodeError / TypeError from inside verify_signature, leaking through verify_and_parse_webhook / _sqs / _sns and breaking the documented contract that says malformed inputs must surface as WebhookSignatureError. The boolean primitive now returns False for those inputs (an invalid-format signature can by definition never match), so the composite helpers raise WebhookSignatureError("invalid webhook signature") as expected. The constant-time HMAC comparison path is unchanged for well-formed inputs. Adds regression tests for non-ASCII bytes, non-ASCII str, and non-string signature inputs at both the primitive and composite layers. Co-authored-by: Cursor <cursoragent@cursor.com> * docs(webhooks): align compression section with the shipped API The previous draft referenced helpers that were renamed during the refactor to the verify_and_parse_* contract (CHA-3071): - client.verify_and_decode_webhook(...) -> verify_and_parse_webhook - decompress_webhook_body(...) -> removed (no public form) - content_encoding / payload_encoding -> removed (magic-byte detect) Following the old snippets would hit AttributeError immediately. The section is rewritten to document the real surface area: - client.verify_and_parse_webhook(body, signature) - client.verify_and_parse_sqs(message_body, signature) - client.verify_and_parse_sns(message, signature) - module-level webhook.verify_and_parse_* helpers for stateless use - WebhookSignatureError as the single error class It also clarifies the return type (parsed dict, not raw bytes) and notes that the legacy verify_webhook bool helper stays unchanged. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(webhooks): unwrap SNS notification envelope in decode_sns_payload decode_sns_payload now JSON-parses the SNS HTTP notification envelope ({"Type":"Notification","Message":"..."}) and extracts the inner Message field before running the SQS pipeline. Falls through to the pre-extracted Message string when the input is not a JSON envelope so existing call sites keep working. Test adds a realistic SNS HTTP notification body fixture and exercises both the new envelope path and the existing pre-extracted Message path. Docs updated to show the typical "pass the raw HTTP body" call site. Co-authored-by: Cursor <cursoragent@cursor.com> * style(webhooks): apply black formatting to SNS envelope test additions Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(webhooks): rename ungzip_payload to gunzip_payload + add golden fixtures (CHA-3071) Per Tommaso's suggestion, align the gzip helper with the GNU `gunzip` command name. The function was added in this PR and not yet released, so this is a straight rename with no back-compat alias. Adds Tommaso's reference fixtures to the test suite as named cases so future SDKs can sanity-check against the same payloads: aGVsbG93b3JsZA== -> helloworld (base64) H4sIAGrYAWoAA8tIzcnJL88vykkBAK0g6/kKAAAA -> helloworld (base64+gzip) Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(webhooks): unify webhook errors under InvalidWebhookError (CHA-3071) Per cross-SDK coordination (mogita's review on the 6 sibling SDK PRs), every webhook failure path now terminates at a single exception class. Customers only need one except arm and can filter by message text for mode-specific behaviour (signature mismatch vs invalid base64 etc.). Renames the previously-unreleased WebhookSignatureError to InvalidWebhookError and threads it through every primitive: verify_signature -> 'signature mismatch' gunzip_payload -> 'gzip decompression failed' decode_sqs_payload -> 'invalid base64 encoding' parse_event -> 'invalid JSON payload' StreamChat#verify_webhook (the legacy bool helper) is untouched. The message constants are exported so callers can exact-match if they prefer that over substring matching. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(webhooks): make signature optional on verify_and_parse_sqs/sns (CHA-3071) Stream does not ship an X-Signature on SQS or SNS deliveries — those transports ride AWS-internal infrastructure (IAM-authenticated queues and AWS-signed SNS notifications), so HMAC verification on top is theatre. signature + secret are now optional on both module helpers and on the StreamChat / StreamChatAsync instance methods. - verify_and_parse_sqs(body) -> decode + parse - verify_and_parse_sqs(body, sig, secret) -> decode + verify + parse - verify_and_parse_sns(envelope_body) -> unwrap + decode + parse - verify_and_parse_sns(envelope_body, sig, secret) -> + verify Passing only one of (signature, secret) raises InvalidWebhookError. The HTTP-webhook path (verify_and_parse_webhook) is unchanged. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(webhooks): parseSqs/ParseSns decode-only; HTTP verify via verifyAndParseWebhook; docs + tests Co-authored-by: Cursor <cursoragent@cursor.com> * fix(webhooks): Go ErrInvalidWebhook + VerifySignature(error); Ruby WebhookSignatureError + parse_*; Python WebhookSignatureError; guard test init without STREAM_* * feat(webhooks): align cross-SDK contract — InvalidWebhookError + gunzip_payload - Rename WebhookSignatureError → InvalidWebhookError - Rename ungzip_payload → gunzip_payload - Align error messages to documented strings: signature mismatch / invalid base64 encoding / gzip decompression failed / invalid JSON payload - parse_event now wraps json.JSONDecodeError as InvalidWebhookError - Export INVALID_WEBHOOK_* constants for exact-match filtering --------- Co-authored-by: Cursor <cursoragent@cursor.com>
1 parent 7640db7 commit 2872cb4

5 files changed

Lines changed: 656 additions & 0 deletions

File tree

docs/webhooks/webhooks_overview/webhooks_overview.md

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,96 @@ valid = client.verify_webhook(request.body, request.META['HTTP_X_SIGNATURE'])
9090
valid = client.verify_webhook(request.data, request.headers['X-SIGNATURE'])
9191
```
9292

93+
### Compressed webhook bodies
94+
95+
GZIP compression can be enabled for hook payloads from the Dashboard. Enabling compression reduces the payload size significantly (often 70–90% smaller) reducing your bandwidth usage on Stream. The decompression cost on your side is usually negligible and offset by the much smaller payload.
96+
97+
When payload compression is enabled, webhook HTTP requests include the `Content-Encoding: gzip` header and the body is gzipped. SQS and SNS messages are gzipped and then base64-wrapped (both transports are UTF-8 only). Some HTTP servers and middleware (Rails, Django, Laravel, Spring Boot, ASP.NET) auto-decompress the body before your handler runs — in that case the body you see is already raw JSON.
98+
99+
Before enabling compression, make sure that:
100+
101+
* Your backend integration is using a recent version of our official SDKs with compression support
102+
* If you don't use an official SDK, make sure that your code supports receiving compressed payloads
103+
* The payload signature check is done on the **uncompressed** payload
104+
105+
The Python SDK exposes a one-liner per transport. Each helper detects the encoding from the body bytes (the gzip magic `1f 8b`, per [RFC 1952](https://datatracker.ietf.org/doc/html/rfc1952)), verifies the HMAC `X-Signature` over the uncompressed JSON, and returns the parsed event as a `dict`. Typed event classes are planned for a future release; until then handlers can key off the `type` field.
106+
107+
```python
108+
from stream_chat import StreamChat
109+
110+
client = StreamChat(api_key="STREAM_KEY", api_secret="STREAM_SECRET")
111+
112+
# Django view
113+
def stream_webhook(request):
114+
event = client.verify_and_parse_webhook(
115+
request.body,
116+
request.headers["X-Signature"],
117+
)
118+
# ... handle event["type"], event["message"], ...
119+
```
120+
121+
```python
122+
from flask import request
123+
from stream_chat import StreamChat
124+
125+
client = StreamChat(api_key="STREAM_KEY", api_secret="STREAM_SECRET")
126+
127+
@app.route("/webhooks/stream", methods=["POST"])
128+
def stream_webhook():
129+
event = client.verify_and_parse_webhook(
130+
request.get_data(),
131+
request.headers["X-Signature"],
132+
)
133+
# ... handle event["type"], event["message"], ...
134+
```
135+
136+
The same call works whether or not Stream is compressing for this app, and whether or not your framework auto-decompressed the request — the helper inspects the body bytes rather than the `Content-Encoding` header.
137+
138+
All helpers raise `stream_chat.webhook.InvalidWebhookError` when the signature does not match, when the gzip stream is corrupt, or when the SQS/SNS base64 envelope cannot be decoded.
139+
140+
The original `client.verify_webhook(request.body, request.headers["X-Signature"])` — which returns a `bool` and does not decompress — stays unchanged for backward compatibility. Switch to `verify_and_parse_webhook` to support compressed payloads.
141+
142+
#### SQS / SNS firehose
143+
144+
For events delivered through SQS or SNS, call the matching helper. It base64-decodes the envelope, gzip-decompresses when the magic bytes are present, and returns the parsed event.
145+
146+
Stream does **not** ship an `X-Signature` on SQS or SNS deliveries: those transports run on AWS-internal infrastructure that is already authenticated end-to-end. SQS queues are reached via IAM-authenticated polling, and SNS notifications carry an AWS signature on the notification envelope itself, so verifying that the message really came from your topic happens at the AWS layer. Layering an HMAC check on top is redundant, so the SQS/SNS helpers only decode and parse — they take a single argument and never verify a signature.
147+
148+
For SQS, pass the message `Body` (already the payload):
149+
150+
```python
151+
event = client.parse_sqs(sqs_message["Body"])
152+
```
153+
154+
For SNS, pass the **raw notification body** (the full `{"Type":"Notification", ...}` JSON envelope Amazon delivers). The SDK extracts the inner `Message` field for you, so the call site mirrors what HTTP frameworks already hand you in `request.body`:
155+
156+
```python
157+
# Django SNS HTTP delivery
158+
event = client.parse_sns(request.body) # raw envelope (bytes/str)
159+
```
160+
161+
#### Stateless / module-level form
162+
163+
If you do not want to construct a `StreamChat` client (for example in a lightweight Lambda that only handles webhooks), call the module-level helpers directly. The HTTP helper still requires the signature and secret; the SQS/SNS helpers take a single argument:
164+
165+
```python
166+
from stream_chat import webhook
167+
168+
event = webhook.verify_and_parse_webhook(body, signature, secret)
169+
event = webhook.parse_sqs(message_body)
170+
event = webhook.parse_sns(notification_body)
171+
```
172+
173+
##### Arguments
174+
175+
| Argument | `verify_and_parse_webhook` | `parse_sqs` | `parse_sns` |
176+
| ------------------- | -------------------------- | --------------- | -------------------- |
177+
| body / message_body / notification_body | required | required | required |
178+
| signature | required |||
179+
| secret | required |||
180+
181+
The module also exposes the primitives the composites are built from — `gunzip_payload`, `decode_sqs_payload`, `decode_sns_payload`, `verify_signature` (constant-time HMAC-SHA256), and `parse_event` — for callers that need to run the steps individually.
182+
93183
All webhook requests contain these headers:
94184

95185
| Name | Description | Example |

stream_chat/base/client.py

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -133,6 +133,41 @@ def verify_webhook(
133133
).hexdigest()
134134
return signature == x_signature
135135

136+
def verify_and_parse_webhook(
137+
self,
138+
body: Union[bytes, str],
139+
signature: Union[str, bytes],
140+
) -> Dict[str, Any]:
141+
"""Verify and parse an HTTP webhook event.
142+
143+
Decompresses ``body`` when gzipped (detected from the body bytes),
144+
verifies the ``X-Signature`` header against the app's API secret,
145+
and returns the parsed event. The Python SDK currently returns a
146+
``dict``; typed event classes are planned for a future release.
147+
148+
:param body: raw HTTP request body bytes Stream signed
149+
:param signature: ``X-Signature`` header value
150+
:raises stream_chat.base.exceptions.InvalidWebhookError: on
151+
signature mismatch or any decode error
152+
"""
153+
from stream_chat.webhook import verify_and_parse_webhook
154+
155+
return verify_and_parse_webhook(body, signature, self.api_secret)
156+
157+
def parse_sqs(self, message_body: Union[bytes, str]) -> Dict[str, Any]:
158+
"""Parse an SQS firehose body (base64 + optional gzip). No HMAC."""
159+
160+
from stream_chat.webhook import parse_sqs
161+
162+
return parse_sqs(message_body)
163+
164+
def parse_sns(self, message: Union[bytes, str]) -> Dict[str, Any]:
165+
"""Parse an SNS body (unwraps SNS envelope when present). No HMAC."""
166+
167+
from stream_chat.webhook import parse_sns
168+
169+
return parse_sns(message)
170+
136171
@abc.abstractmethod
137172
def update_app_settings(
138173
self, **settings: Any

stream_chat/base/exceptions.py

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,23 @@ class StreamChannelException(Exception):
66
pass
77

88

9+
class InvalidWebhookError(Exception):
10+
"""Invalid webhook signature or malformed gzip/base64/JSON envelope.
11+
12+
Raised by :mod:`stream_chat.webhook` on any failure path: signature
13+
mismatch, malformed base64, gzip decompression failure, or invalid
14+
JSON payload. The message text identifies the failure mode so
15+
callers that want to differentiate (security logging, retry policy)
16+
can filter on substring or on the module-level constants.
17+
"""
18+
19+
20+
INVALID_WEBHOOK_SIGNATURE_MISMATCH = "signature mismatch"
21+
INVALID_WEBHOOK_INVALID_BASE64 = "invalid base64 encoding"
22+
INVALID_WEBHOOK_GZIP_FAILED = "gzip decompression failed"
23+
INVALID_WEBHOOK_INVALID_JSON = "invalid JSON payload"
24+
25+
926
class StreamAPIException(Exception):
1027
def __init__(self, text: str, status_code: int) -> None:
1128
self.response_text = text

0 commit comments

Comments
 (0)