Skip to content

Commit f9d11fc

Browse files
KyleAMathewsclaudesamwillis
committed
feat: add expression helpers for parsing LoadSubsetOptions in queryFn (#786)
* feat: add expression helpers for parsing LoadSubsetOptions in queryFn Add reusable expression parsing utilities to help developers translate TanStack DB predicates (where, orderBy, limit) into their API's format. - Add expression-helpers.ts with generic parsing utilities - parseWhereExpression: Parse where clauses with custom handlers - parseOrderByExpression: Parse order by into simple array - extractSimpleComparisons: Extract simple AND-ed filters - parseLoadSubsetOptions: Convenience function for all options - walkExpression, extractFieldPath, extractValue: Lower-level helpers - Export helpers from query-db-collection package - Add comprehensive documentation to query-collection.md - How LoadSubsetOptions are passed via ctx.meta - Expression helper usage with REST and GraphQL examples - API reference for all helper functions - Tips and best practices This makes it much easier to implement query collections with predicate push-down without having to manually parse expression AST trees. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * test: add comprehensive tests for expression helpers - 35 test cases covering all helper functions - Tests for parseWhereExpression with various operators - Tests for parseOrderByExpression with sorting options - Tests for extractSimpleComparisons - Tests for parseLoadSubsetOptions - Tests for low-level helpers (extractFieldPath, extractValue, walkExpression) - Integration test for complex real-world query scenarios All tests passing with 93.65% coverage of expression-helpers.ts 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * chore: add changeset for expression helpers 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * docs: make GraphQL example more generic Remove Hasura-specific emphasis in GraphQL example section. The underscore-prefixed operators are common GraphQL conventions. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * docs: clarify GraphQL format is not prescriptive Change 'the GraphQL format' to 'a GraphQL format' to avoid implying this is the only way to structure GraphQL queries. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * docs: remove Electric DB reference from tips Remove implementation-specific reference to Electric DB Collection from the tips section. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: make loadSubsetOptions always an object - Add default empty object parameter to createQueryFromOpts - Remove ?? {} pattern from all documentation - Update changeset example - Update PR description This makes the API cleaner - users can now access ctx.meta.loadSubsetOptions directly without null coalescing. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: add explicit type annotations to avoid implicit any - Import BasicExpression and OrderBy from IR namespace - Add type annotations to all lambda parameters - Remove unnecessary type checks that were always true - Fix variable shadowing in walkExpression visitor parameter - Fixes build type errors and eslint warnings in expression-helpers.ts 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * style: fix prettier formatting in changeset 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: correct TypeScript types in expression-helpers tests * feat: address Sam's review feedback - Add OperatorName type and operators constant to db package - Improve ParseWhereOptions handlers type to know IR operators - Support array indices in FieldPath (string | number) - Add locale/string collation options to ParsedOrderBy - Make extractSimpleComparisons throw on unsupported operations (or, not, etc.) - Update tests to expect throws instead of silent skipping * fix: remove invalid sensitivity/locale fields from test mocks CompareOptions uses StringCollationConfig which has stringSort, locale, and localeOptions fields, not sensitivity and locale at the top level. * fix: handle discriminated union for CompareOptions properly Use 'in' operator to check for optional fields (locale, localeOptions) that only exist when stringSort is 'locale' * move helpers to main db package --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Sam Willis <sam.willis@gmail.com>
1 parent 243a35a commit f9d11fc

22 files changed

Lines changed: 2404 additions & 2 deletions
Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
---
2+
"@tanstack/db": patch
3+
"@tanstack/query-db-collection": patch
4+
---
5+
6+
Add expression helper utilities for parsing LoadSubsetOptions in queryFn.
7+
8+
When using `syncMode: 'on-demand'`, TanStack DB now provides helper functions to easily parse where clauses, orderBy, and limit predicates into your API's format:
9+
10+
- `parseWhereExpression`: Parse where clauses with custom handlers for each operator
11+
- `parseOrderByExpression`: Parse order by into simple array format
12+
- `extractSimpleComparisons`: Extract simple AND-ed filters
13+
- `parseLoadSubsetOptions`: Convenience function to parse all options at once
14+
- `walkExpression`, `extractFieldPath`, `extractValue`: Lower-level helpers
15+
16+
**Example:**
17+
18+
```typescript
19+
import { parseLoadSubsetOptions } from "@tanstack/db"
20+
// or from "@tanstack/query-db-collection" (re-exported for convenience)
21+
22+
queryFn: async (ctx) => {
23+
const { where, orderBy, limit } = ctx.meta.loadSubsetOptions
24+
25+
const parsed = parseLoadSubsetOptions({ where, orderBy, limit })
26+
27+
// Build API request from parsed filters
28+
const params = new URLSearchParams()
29+
parsed.filters.forEach(({ field, operator, value }) => {
30+
if (operator === "eq") {
31+
params.set(field.join("."), String(value))
32+
}
33+
})
34+
35+
return fetch(`/api/products?${params}`).then((r) => r.json())
36+
}
37+
```
38+
39+
This eliminates the need to manually traverse expression AST trees when implementing predicate push-down.

docs/collections/query-collection.md

Lines changed: 306 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -398,3 +398,309 @@ All direct write methods are available on `collection.utils`:
398398
- `writeUpsert(data)`: Insert or update one or more items directly
399399
- `writeBatch(callback)`: Perform multiple operations atomically
400400
- `refetch(opts?)`: Manually trigger a refetch of the query
401+
402+
## QueryFn and Predicate Push-Down
403+
404+
When using `syncMode: 'on-demand'`, the collection automatically pushes down query predicates (where clauses, orderBy, and limit) to your `queryFn`. This allows you to fetch only the data needed for each specific query, rather than fetching the entire dataset.
405+
406+
### How LoadSubsetOptions Are Passed
407+
408+
LoadSubsetOptions are passed to your `queryFn` via the query context's `meta` property:
409+
410+
```typescript
411+
queryFn: async (ctx) => {
412+
// Extract LoadSubsetOptions from the context
413+
const { limit, where, orderBy } = ctx.meta.loadSubsetOptions
414+
415+
// Use these to fetch only the data you need
416+
// ...
417+
}
418+
```
419+
420+
The `where` and `orderBy` fields are expression trees (AST - Abstract Syntax Tree) that need to be parsed. TanStack DB provides helper functions to make this easy.
421+
422+
### Expression Helpers
423+
424+
```typescript
425+
import {
426+
parseWhereExpression,
427+
parseOrderByExpression,
428+
extractSimpleComparisons,
429+
parseLoadSubsetOptions,
430+
} from '@tanstack/db'
431+
// Or from '@tanstack/query-db-collection' (re-exported for convenience)
432+
```
433+
434+
These helpers allow you to parse expression trees without manually traversing complex AST structures.
435+
436+
### Quick Start: Simple REST API
437+
438+
```typescript
439+
import { createCollection } from '@tanstack/react-db'
440+
import { queryCollectionOptions } from '@tanstack/query-db-collection'
441+
import { parseLoadSubsetOptions } from '@tanstack/db'
442+
import { QueryClient } from '@tanstack/query-core'
443+
444+
const queryClient = new QueryClient()
445+
446+
const productsCollection = createCollection(
447+
queryCollectionOptions({
448+
id: 'products',
449+
queryKey: ['products'],
450+
queryClient,
451+
getKey: (item) => item.id,
452+
syncMode: 'on-demand', // Enable predicate push-down
453+
454+
queryFn: async (ctx) => {
455+
const { limit, where, orderBy } = ctx.meta.loadSubsetOptions
456+
457+
// Parse the expressions into simple format
458+
const parsed = parseLoadSubsetOptions({ where, orderBy, limit })
459+
460+
// Build query parameters from parsed filters
461+
const params = new URLSearchParams()
462+
463+
// Add filters
464+
parsed.filters.forEach(({ field, operator, value }) => {
465+
const fieldName = field.join('.')
466+
if (operator === 'eq') {
467+
params.set(fieldName, String(value))
468+
} else if (operator === 'lt') {
469+
params.set(`${fieldName}_lt`, String(value))
470+
} else if (operator === 'gt') {
471+
params.set(`${fieldName}_gt`, String(value))
472+
}
473+
})
474+
475+
// Add sorting
476+
if (parsed.sorts.length > 0) {
477+
const sortParam = parsed.sorts
478+
.map(s => `${s.field.join('.')}:${s.direction}`)
479+
.join(',')
480+
params.set('sort', sortParam)
481+
}
482+
483+
// Add limit
484+
if (parsed.limit) {
485+
params.set('limit', String(parsed.limit))
486+
}
487+
488+
const response = await fetch(`/api/products?${params}`)
489+
return response.json()
490+
},
491+
})
492+
)
493+
494+
// Usage with live queries
495+
import { createLiveQueryCollection } from '@tanstack/react-db'
496+
import { eq, lt, and } from '@tanstack/db'
497+
498+
const affordableElectronics = createLiveQueryCollection({
499+
query: (q) =>
500+
q.from({ product: productsCollection })
501+
.where(({ product }) => and(
502+
eq(product.category, 'electronics'),
503+
lt(product.price, 100)
504+
))
505+
.orderBy(({ product }) => product.price, 'asc')
506+
.limit(10)
507+
.select(({ product }) => product)
508+
})
509+
510+
// This triggers a queryFn call with:
511+
// GET /api/products?category=electronics&price_lt=100&sort=price:asc&limit=10
512+
```
513+
514+
### Custom Handlers for Complex APIs
515+
516+
For APIs with specific formats, use custom handlers:
517+
518+
```typescript
519+
queryFn: async (ctx) => {
520+
const { where, orderBy, limit } = ctx.meta.loadSubsetOptions
521+
522+
// Use custom handlers to match your API's format
523+
const filters = parseWhereExpression(where, {
524+
handlers: {
525+
eq: (field, value) => ({
526+
field: field.join('.'),
527+
op: 'equals',
528+
value
529+
}),
530+
lt: (field, value) => ({
531+
field: field.join('.'),
532+
op: 'lessThan',
533+
value
534+
}),
535+
and: (...conditions) => ({
536+
operator: 'AND',
537+
conditions
538+
}),
539+
or: (...conditions) => ({
540+
operator: 'OR',
541+
conditions
542+
}),
543+
}
544+
})
545+
546+
const sorts = parseOrderByExpression(orderBy)
547+
548+
return api.query({
549+
filters,
550+
sort: sorts.map(s => ({
551+
field: s.field.join('.'),
552+
order: s.direction.toUpperCase()
553+
})),
554+
limit
555+
})
556+
}
557+
```
558+
559+
### GraphQL Example
560+
561+
```typescript
562+
queryFn: async (ctx) => {
563+
const { where, orderBy, limit } = ctx.meta.loadSubsetOptions
564+
565+
// Convert to a GraphQL where clause format
566+
const whereClause = parseWhereExpression(where, {
567+
handlers: {
568+
eq: (field, value) => ({
569+
[field.join('_')]: { _eq: value }
570+
}),
571+
lt: (field, value) => ({
572+
[field.join('_')]: { _lt: value }
573+
}),
574+
and: (...conditions) => ({ _and: conditions }),
575+
or: (...conditions) => ({ _or: conditions }),
576+
}
577+
})
578+
579+
// Convert to a GraphQL order_by format
580+
const sorts = parseOrderByExpression(orderBy)
581+
const orderByClause = sorts.map(s => ({
582+
[s.field.join('_')]: s.direction
583+
}))
584+
585+
const { data } = await graphqlClient.query({
586+
query: gql`
587+
query GetProducts($where: product_bool_exp, $orderBy: [product_order_by!], $limit: Int) {
588+
product(where: $where, order_by: $orderBy, limit: $limit) {
589+
id
590+
name
591+
category
592+
price
593+
}
594+
}
595+
`,
596+
variables: {
597+
where: whereClause,
598+
orderBy: orderByClause,
599+
limit
600+
}
601+
})
602+
603+
return data.product
604+
}
605+
```
606+
607+
### Expression Helper API Reference
608+
609+
#### `parseLoadSubsetOptions(options)`
610+
611+
Convenience function that parses all LoadSubsetOptions at once. Good for simple use cases.
612+
613+
```typescript
614+
const { filters, sorts, limit } = parseLoadSubsetOptions(ctx.meta?.loadSubsetOptions)
615+
// filters: [{ field: ['category'], operator: 'eq', value: 'electronics' }]
616+
// sorts: [{ field: ['price'], direction: 'asc', nulls: 'last' }]
617+
// limit: 10
618+
```
619+
620+
#### `parseWhereExpression(expr, options)`
621+
622+
Parses a WHERE expression using custom handlers for each operator. Use this for complete control over the output format.
623+
624+
```typescript
625+
const filters = parseWhereExpression(where, {
626+
handlers: {
627+
eq: (field, value) => ({ [field.join('.')]: value }),
628+
lt: (field, value) => ({ [`${field.join('.')}_lt`]: value }),
629+
and: (...filters) => Object.assign({}, ...filters)
630+
},
631+
onUnknownOperator: (operator, args) => {
632+
console.warn(`Unsupported operator: ${operator}`)
633+
return null
634+
}
635+
})
636+
```
637+
638+
#### `parseOrderByExpression(orderBy)`
639+
640+
Parses an ORDER BY expression into a simple array.
641+
642+
```typescript
643+
const sorts = parseOrderByExpression(orderBy)
644+
// Returns: [{ field: ['price'], direction: 'asc', nulls: 'last' }]
645+
```
646+
647+
#### `extractSimpleComparisons(expr)`
648+
649+
Extracts simple AND-ed comparisons from a WHERE expression. Note: Only works for simple AND conditions.
650+
651+
```typescript
652+
const comparisons = extractSimpleComparisons(where)
653+
// Returns: [
654+
// { field: ['category'], operator: 'eq', value: 'electronics' },
655+
// { field: ['price'], operator: 'lt', value: 100 }
656+
// ]
657+
```
658+
659+
### Supported Operators
660+
661+
- `eq` - Equality (=)
662+
- `gt` - Greater than (>)
663+
- `gte` - Greater than or equal (>=)
664+
- `lt` - Less than (<)
665+
- `lte` - Less than or equal (<=)
666+
- `and` - Logical AND
667+
- `or` - Logical OR
668+
- `in` - IN clause
669+
670+
### Using Query Key Builders
671+
672+
Create different cache entries for different filter combinations:
673+
674+
```typescript
675+
const productsCollection = createCollection(
676+
queryCollectionOptions({
677+
id: 'products',
678+
// Dynamic query key based on filters
679+
queryKey: (opts) => {
680+
const parsed = parseLoadSubsetOptions(opts)
681+
const cacheKey = ['products']
682+
683+
parsed.filters.forEach(f => {
684+
cacheKey.push(`${f.field.join('.')}-${f.operator}-${f.value}`)
685+
})
686+
687+
if (parsed.limit) {
688+
cacheKey.push(`limit-${parsed.limit}`)
689+
}
690+
691+
return cacheKey
692+
},
693+
queryClient,
694+
getKey: (item) => item.id,
695+
syncMode: 'on-demand',
696+
queryFn: async (ctx) => { /* ... */ },
697+
})
698+
)
699+
```
700+
701+
### Tips
702+
703+
1. **Start with `parseLoadSubsetOptions`** for simple use cases
704+
2. **Use custom handlers** via `parseWhereExpression` for APIs with specific formats
705+
3. **Handle unsupported operators** with the `onUnknownOperator` callback
706+
4. **Log parsed results** during development to verify correctness
Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
---
2+
id: extractFieldPath
3+
title: extractFieldPath
4+
---
5+
6+
# Function: extractFieldPath()
7+
8+
```ts
9+
function extractFieldPath(expr): FieldPath | null;
10+
```
11+
12+
Defined in: [packages/db/src/query/expression-helpers.ts:107](https://github.com/TanStack/db/blob/main/packages/db/src/query/expression-helpers.ts#L107)
13+
14+
Extracts the field path from a PropRef expression.
15+
Returns null for non-ref expressions.
16+
17+
## Parameters
18+
19+
### expr
20+
21+
`BasicExpression`
22+
23+
The expression to extract from
24+
25+
## Returns
26+
27+
[`FieldPath`](../../type-aliases/FieldPath.md) \| `null`
28+
29+
The field path array, or null
30+
31+
## Example
32+
33+
```typescript
34+
const field = extractFieldPath(someExpression)
35+
// Returns: ['product', 'category']
36+
```

0 commit comments

Comments
 (0)