Skip to content

feat(codegen/go): PoC for opt-in iter.Seq2 companions on :many queries#4488

Open
Tilzen wants to merge 2 commits into
sqlc-dev:mainfrom
Tilzen:feat/emit-iterators-poc
Open

feat(codegen/go): PoC for opt-in iter.Seq2 companions on :many queries#4488
Tilzen wants to merge 2 commits into
sqlc-dev:mainfrom
Tilzen:feat/emit-iterators-poc

Conversation

@Tilzen

@Tilzen Tilzen commented Jun 14, 2026

Copy link
Copy Markdown

Why

Revives the long-standing request for streaming/lazy :many query results (#720). Today, sqlc always materializes :many into []T, which is fine for small result sets but costly for exports, sync jobs, and backfills.

Go 1.23+ shipped native iterators (iter.Seq2). Kyle noted that this unblocks native iterator generation; PR #3631 explored an implementation but closed without merge after API design remained open.

This PR is not a merge-ready feature. It is a proof of concept (PoC) posted for design feedback before investing in a production implementation. See the discussion proposal in #720 and related #4464.

PoC reference (fork): https://github.com/Tilzen/sqlc/tree/feat/emit-iterators-poc
PRD in PoC branch: docs/emit-iterators-prd.md

What Changed (PoC scope)

Opt-in config (zero breaking changes to default :many[]T):

gen:
  go:
    emit_iterators: true
    iterator_scope: global         # or explicit_only
    iterator_method_prefix: "Iter"
    iterator_style: seq2           # MVP: seq2 only
    iterator_start: lazy

Generates a lazy companion alongside existing slice methods:

func (q *Queries) ListAuthors(ctx context.Context) ([]Author, error) { /* unchanged */ }

func (q *Queries) IterAuthors(ctx context.Context) iter.Seq2[Author, error] {
    return func(yield func(Author, error) bool) {
        rows, err := q.db.QueryContext(ctx, listAuthors)
        // defer rows.Close(), yield rows one at a time
    }
}

Also supports query annotations for explicit_only scope:

  • -- name: StreamAuthors :stream
  • -- name: ListAuthors :many:stream

PoC includes: stdlib + pgx/v5 templates, metadata parsing, config wiring, PRD doc.

Explicitly out of PoC scope: callback/rows styles, eager mode, emit_interface iter methods, Python/Kotlin generators, full driver matrix.

Open Design Questions (feedback welcome)

  1. Method prefix: Iter vs Stream?
  2. Default lazy start — agree?
  3. Ship both emit_iterators: global and :stream annotations?
  4. Should :stream remain a separate query kind or only :many:stream?
  5. Min Go version bump when enabled?

Test Plan

PoC test coverage (all passing on the fork branch):

  • Unit: iterator naming, scope logic, stream annotation parsing, opts validation
  • Codegen matrix: global, explicit_only (:stream, :many:stream), emit_iterators: false regression, parameterized :many, pgx/v5
  • Golden replay: TestReplay/emit_iterators* fixtures
  • Runtime (SQLite): iter/slice parity, lazy start (error only on first range), early break + connection reuse, parameterized queries
go test ./internal/codegen/golang/... ./internal/metadata/... \
  ./internal/endtoend/ -run 'EmitIterators|TestReplay/emit_iterators|ValidateOptsEmitIterators|Iterator'

cd internal/endtoend/testdata
go test ./emit_iterators/... ./emit_iterators_params/... -v

References

Notes

  • This is a PoC for discussion, not a request to merge as-is. Happy to close this PR and iterate on design in Ability to return an iterator on a "many" query #720 if that is preferred.
  • If maintainers sign off on the config surface + default API (seq2 + lazy + Iter prefix), I can refine the PoC toward a production-ready PR in follow-up commits.

Tilzen added 2 commits June 14, 2026 16:09
Implement emit_iterators PoC from docs/emit-iterators-prd.md: lazy
iter.Seq2[T, error] methods alongside existing slice APIs, with global
or explicit_only scope and :stream / :many:stream query annotations.

Includes stdlib and pgx templates, config options, end-to-end testdata,
and codegen integration tests.
…rix tests

Add SQLite runtime tests (parity, lazy start, early break, parameterized
queries), codegen matrix for global/explicit/pgx/disabled scenarios,
opts validation tests, and golden replay fixtures for all cases.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant