Polars Nexpresso is a utility library for working with nested and hierarchical data in Polars. It provides two main capabilities:
- Nested Expression Builder - Clean, intuitive syntax for transforming deeply nested structs and lists
- Hierarchical Packer - Pack/unpack operations for hierarchical data, similar to pandas MultiIndex but using Polars' native nested types
Nexpresso = Nested Expression + ☕ (espresso)
pip install polars-nexpressoOr using uv:
uv add polars-nexpressoTransform deeply nested data with intuitive dictionary syntax:
import polars as pl
from nexpresso import generate_nested_exprs
df = pl.DataFrame({
"order": [
{"customer": "Alice", "items": [{"name": "Laptop", "price": 999}, {"name": "Mouse", "price": 25}]},
{"customer": "Bob", "items": [{"name": "Keyboard", "price": 75}]},
]
})
# Define transformations declaratively
fields = {
"order": {
"items": {
"price": lambda x: x * 1.1, # 10% price increase
"discounted": pl.field("price") * 0.9, # New field
}
}
}
exprs = generate_nested_exprs(fields, df.schema, struct_mode="with_fields")
result = df.select(exprs)Build and navigate hierarchical data from normalized tables:
from nexpresso import HierarchicalPacker, HierarchySpec, LevelSpec
# Define hierarchy
spec = HierarchySpec.from_levels(
LevelSpec(name="region", id_fields=["id"]),
LevelSpec(name="store", id_fields=["id"], parent_keys=["region_id"]),
)
packer = HierarchicalPacker(spec)
# Build from separate tables (like database tables)
nested = packer.build_from_tables({
"region": regions_df,
"store": stores_df,
})
# Navigate between granularities. The level names the granularity you want,
# for both directions: one row per store, then one row per region.
flat = packer.unpack(nested, "store") # Explode to store level
packed = packer.pack(flat, "region") # Aggregate back to region level| Feature | Description |
|---|---|
| Field Selection | Keep fields as-is with None |
| Transformations | Apply lambdas: lambda x: x * 2 |
| New Fields | Create with pl.Expr: pl.field("a") + pl.field("b") |
| Deep Nesting | Works with any depth of structs/lists |
| Two Modes | select (keep specified) or with_fields (keep all) |
| Feature | Description |
|---|---|
| Build from Tables | Join normalized tables into nested hierarchy |
| Pack/Unpack | Navigate between granularity levels |
| Multiple Branches | A level can carry several independent child branches; pack/unpack along either axis |
| Streaming Pack/Unpack | Memory-bounded pack_streaming / unpack_streaming for large data |
| Normalize/Denormalize | Split into per-level tables and reconstruct |
| Validation | Check for null keys and data integrity |
| Custom Separators | Use any separator (default: .) |
| Type Preservation | DataFrame in = DataFrame out |
Query normalized per-level storage through a nested interface.
| Feature | Description |
|---|---|
| A frame per granularity | view.level("sale") returns a LazyFrame; everything after it is plain Polars |
| Cross-level expressions | Combine leaf, parent and grandparent columns in one expression — impossible inside list.eval |
| No hand-written joins | level() joins the axis, and the planner prunes what you do not read |
| Transitive key pushdown | Ancestor-key predicates reach the deepest scan with no join |
| Deferred consistency | filter restricts the whole hierarchy, applied once at materialization |
| Transform a level, keep the view | with_level() returns a view, so filter, nested and sink_parquet still apply — and promote= lands a roll-up on an ancestor |
| Nest only at the boundary | nested() when a consumer genuinely needs List[Struct] |
When defining transformations:
None: Keep the field unchangeddict: Recursively process nested structuresCallable: Apply function to field (e.g.,lambda x: x * 2)pl.Expr: Create/modify field with full expression
"select": Only keep fields specified in the dictionary"with_fields": Keep all fields, add/modify specified ones
Define your data hierarchy with LevelSpec:
LevelSpec(
name="store", # Level identifier
id_fields=["id"], # Unique key columns
parent_keys=["region_id"], # Foreign key to parent (for build_from_tables)
parent=None, # Parent level name; None = the level declared before
)Levels form a chain by default. Naming a parent explicitly makes the hierarchy
a tree, so one level can carry several independent child branches — a city has
streets (which have buildings) and, orthogonally, services:
spec = HierarchySpec.from_levels(
LevelSpec(name="country", id_fields=["code"]),
LevelSpec(name="city", id_fields=["id"], parent="country", parent_keys=["code"]),
LevelSpec(name="street", id_fields=["id"], parent="city", parent_keys=["city_id"]),
LevelSpec(name="building", id_fields=["id"], parent="street", parent_keys=["street_id"]),
LevelSpec(name="service", id_fields=["kind"], parent="city", parent_keys=["city_id"]),
)Each root → level chain is an axis. A flat frame holds one granularity, so
pack and unpack work along the axis their target names and leave sibling
branches packed:
packer.unpack(nested, "building") # street axis; `country.city.service` stays nested
packer.unpack(nested, "service") # service axis; `country.city.street` stays nestedNothing is dropped and nothing is cross-joined, so re-packing either frame
reproduces the original. HierarchyView follows the same rule: level(g)
joins one axis, and a filter on one branch cascades to the other through their
shared ancestor. See
Hierarchical Data.
df = pl.DataFrame({
"items": [[{"name": "A", "qty": 5}, {"name": "B", "qty": 3}]]
})
fields = {
"items": {
"qty": lambda x: x * 2,
"total": pl.field("qty") * 10,
}
}
result = apply_nested_operations(df, fields, struct_mode="with_fields")fields = {
"customer": {
"discount": pl.when(pl.field("tier") == "Gold")
.then(0.15)
.when(pl.field("tier") == "Silver")
.then(0.10)
.otherwise(0.05),
}
}# Tables with foreign key relationships
regions = pl.DataFrame({"id": ["west", "east"], "name": ["West", "East"]})
stores = pl.DataFrame({
"id": ["s1", "s2"],
"name": ["Store 1", "Store 2"],
"region_id": ["west", "east"]
})
spec = HierarchySpec.from_levels(
LevelSpec(name="region", id_fields=["id"]),
LevelSpec(name="store", id_fields=["id"], parent_keys=["region_id"]),
)
packer = HierarchicalPacker(spec)
nested = packer.build_from_tables({"region": regions, "store": stores})
# Result: Stores nested within their regionsnormalize / split_levels emit one level-local table per level: the
level's own id fields and attributes, plus the key columns of its ancestors as
foreign keys. Coarser attributes are not duplicated into the finer tables, and
descendant columns are never included.
# Split nested data into separate tables
tables = packer.normalize(nested_df)
# region: region.id, region.name
# store : region.id, region.store.id, region.store.revenue
# ^ foreign key ^ own columns only
# Reconstruct from separate tables (round-trips back to the nested frame)
rebuilt = packer.denormalize(tables)pack builds the nested result with a group_by that collects children into
list columns. Polars' streaming engine has no native list-collecting
aggregation, so that node falls back to the in-memory engine and peak memory
scales with the whole dataset. (unpack — explode + unnest — does stream
natively; see Lazy Evaluation & Streaming
for the full breakdown.)
For datasets that don't fit comfortably in RAM, pack_streaming buckets
the input by the root-level key (keeping each entity's rows together), packs
each bucket independently while sinking to Parquet, and returns a chainable
LazyFrame — bounding peak memory to a single bucket.
# Bound peak memory by processing the data in root-key buckets.
# Accepts a DataFrame, LazyFrame, or a Parquet path/glob (scanned lazily).
packed = packer.pack_streaming(flat_df, "region", partitions=32)
# It returns a LazyFrame, so you can keep composing lazily:
top = (
packer.pack_streaming("s3_dump/*.parquet", "region", partitions=64)
.filter(pl.col("region.id").is_in(active_regions))
.collect(engine="streaming")
)
# defer=False sinks eagerly and returns a scan handle so downstream work streams
# straight from disk — safest when even the packed result is too big for RAM.
handle = packer.pack_streaming(flat_df, "region", partitions=64, defer=False)
# unpack already streams; unpack_streaming keeps it lazy / disk-to-disk.
leaves = packer.unpack_streaming("packed.parquet", "store", sink_path="leaves.parquet")More buckets means lower peak memory (and more temporary files). See
benchmarks/ for a peak-RSS comparison of pack vs
pack_streaming.
When the root level carries heavy attributes that repeat across every leaf row
(e.g. a per-entity blob, thumbnail, or embedding), carrying them through the pack
aggregation is wasteful. pack(..., parent_strategy="split_join") instead pulls
those attributes into a small dimension table (unique per root key) and reattaches
them after packing — identical results, but the heavy column is touched once per
entity instead of once per leaf row.
# Equivalent to the default pack, but far cheaper when root attributes dominate.
packed = packer.pack(flat_df, "region", parent_strategy="split_join")This wins big when root attributes are heavy relative to the child data (measured
up to ~9x faster, half the memory); for child-dominated data it adds join overhead
for no gain, so it is opt-in. See benchmarks/ for the trade-offs.
Note on ordering: packing no longer performs a global sort, so top-level row order is not guaranteed. Child-list order is still preserved when
preserve_child_order=True(the default) or via a level'sorder_by. De-duplication and null handling of parent attributes are unaffected.
Generate Polars expressions for nested data operations.
Parameters:
fields: Dictionary defining operations on columns/fieldsschema: DataFrame schema (or DataFrame/LazyFrame to extract schema)struct_mode:"select"or"with_fields"
Returns: list[pl.Expr]
Apply nested operations directly to a DataFrame.
HierarchicalPacker(spec, *, granularity_separator=".", escape_char="\\", preserve_child_order=True, validate_on_pack=True)
Main class for hierarchical operations.
Key Methods:
pack(frame, at_level, *, extra_columns="preserve", parent_strategy="aggregate")- Pack so each row is oneat_levelentity, nesting everything below it (parent_strategy="split_join"reattaches heavy root attributes via a join)pack_streaming(source, at_level, *, partitions=16, partition_strategy="balanced", ...)- Memory-bounded pack for large data ("balanced"buckets by row count and returns root-key-sorted output;"hash"is one pass cheaper but balances entities, not rows)unpack(frame, at_level)- Unpack so each row is oneat_levelentity — the exact inverse ofpackat the same levelnormalize(frame)- Split into per-level tablesdenormalize(tables)- Reconstruct from per-level tables; a true inverse ofnormalize, sodenormalize(normalize(df, at_level=L), at_level=L) == pack(df, L)build_from_tables(tables)- Build hierarchy from normalized tablesvalidate(frame)- Check data integrity
Create a hierarchy specification from level definitions.
key_aliases is deprecated — rename the column on the frame instead
(df.with_columns(pl.col("country.city.id").alias("country.code"))). Synthesised
keys are stripped from the per-level tables, so a hierarchy relying on them cannot
round-trip through normalize / denormalize.
Define a single level in the hierarchy. parent names this level's parent; leave
it None for a linear chain, or set it on every non-root level to branch.
# Run comprehensive examples
python examples.py
# Or run specific module examples
python -m nexpresso.hierarchical_packerList[Struct] is a good in-memory shape and a poor storage shape. Parquet
shreds nested columns into one column chunk per leaf, but a row group holds N
top-level rows — so packing collapses row-group skipping, the main reason
Parquet is fast. On a 2M-row three-level hierarchy, querying the packed file is
30-196x slower than the same data stored flat, and the packed file is only
~15% smaller.
Store flat or normalized; pack at the boundary where something consumes nesting.
HierarchyView makes the normalized layout ergonomic:
from nexpresso import HierarchyView
# One-time conversion.
HierarchyView.from_frame(flat_df, packer).sink_parquet("warehouse/")
# Every subsequent query scans one Parquet dataset per level.
view = HierarchyView.scan_parquet("warehouse/", packer)
hot = view.filter(pl.col("region.store.sale.amount") > 990)
hot.tables()["sale"] # cheapest: no join, no nesting
hot.level("sale") # a LazyFrame at leaf granularity — plain Polars from here
hot.nested().collect() # the packed List[Struct] shapeIt also unlocks something list.eval cannot express at all — Polars rejects
named columns inside an eval context, so a leaf value can never be combined with
a parent attribute in a packed frame. On a view it is just an expression:
view.level("sale").with_columns(
(
pl.col("region.store.sale.amount")
* (1 - pl.col("region.store.discount")) # parent
* (1 + pl.col("region.tax_rate")) # grandparent
).alias("final")
)See Storage Layouts for the measurements,
python examples_hierarchy_view.py for a runnable tour,
python examples_hierarchy_view_recipes.py for the cookbook, and
benchmarks/bench_storage.py to reproduce the numbers.
Both components generate native Polars expressions, so performance is equivalent to hand-written code. All operations are lazy-compatible and benefit from Polars' query optimization.
MIT License - see LICENSE file for details.
Contributions welcome! Please feel free to submit issues and pull requests.