Skip to content

Add helix-shared-queue base package (pluggable queue abstraction) #1269

Description

@tripodsan

Why

helix-api-service (and likely other Helix services) send messages to AWS SQS via BatchedQueueClient (@adobe/helix-admin-support), which hardcodes @aws-sdk/client-sqs + @aws-sdk/client-s3 internally. There's no way to back it with a different queue provider (e.g. Azure Service Bus) without forking that class.

@adobe/helix-shared-storage already solves this exact problem for blob storage: a thin base package with a pluggable backendFactory, consumed today via @adobe/helix-shared-storage-s3. This issue proposes the same pattern for queues.

Proposed interface

Mirrors Storage/Bucket:

  • Queue.fromContext(context, opts) — caches on context.attributes.queue, calls new this(...) so subclasses compose correctly (see Storage.fromContext).
  • Queue#queue(queueName, opts) -> QueueRef, constructed via an injected backendFactory (no cloud SDK in this package at all).
  • QueueRef#send(messages)messages: [{ body, groupId?, dedupId? }] -> { messageIds }. groupId/dedupId are named generically (not MessageGroupId/MessageDeduplicationId) so they can also map onto Azure Service Bus sessionId + native dedup later.
  • QueueRef#receive(opts) / QueueRef#delete(messages) — generalizes BatchedQueueClient.receive()/delete() (long-poll semantics, batch ack), for consumers of these queues outside helix-api-service.
  • AbstractQueueBackend — the contract concrete packages implement (sendBatch/receiveBatch/deleteBatch). Batching/chunking limits are provider-specific (SQS's ≤10 entries/≤256KB vs. Azure Service Bus's dynamic batch sizing), so each backend owns its own batching loop; QueueRef is a thin pass-through, not a shared generic chunker.

Notable design choice: reuse @adobe/helix-shared-storage for oversized-message spillover

BatchedQueueClient hardcodes S3 (PutObjectCommand) for its "message too big for the queue, swap to blob storage" fallback. Backends built on this package should instead accept an injected Storage/Bucket (from @adobe/helix-shared-storage) for that fallback, so the spill-storage choice tracks whatever Storage backend the host service already configured, rather than each queue backend hardcoding its own cloud SDK for a secondary concern.

Out of scope here

  • No concrete backend (see companion issue for helix-shared-queue-sqs).
  • Consumer-side "trigger a function from a queue" wiring (e.g. Lambda SQS event sources) is a runtime/deployment concern, not part of this package.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions