Skip to content

Home

Jump to bottom
Sergio Soto edited this page Apr 8, 2026 · 3 revisions

oiio-proxy-generator

Serverless thumbnail and proxy generation for high-resolution media on VAST DataEngine.

oiio-proxy-generator is a VAST DataEngine function that automatically generates thumbnails and H.264 proxies from EXR, DPX, and other high-resolution media files as they are ingested into a VAST S3 bucket. It applies color space transforms using oiiotool's built-in color management, persists generation metadata to VAST DataBase, and publishes completion events via Kafka.

How It Works

EXR/DPX file uploaded to S3 bucket
  --> VAST DataEngine ElementCreated trigger (.exr/.dpx suffix filter)
    --> oiio-proxy-generator function container
      --> S3 GET source file
      --> Detect color space from EXR metadata
      --> oiiotool colorconvert linear -> sRGB, resize 256x256 -> JPEG thumbnail
      --> oiiotool colorconvert linear -> Rec709, resize 1920x1080 -> ffmpeg -> H.264 MP4 proxy
      --> S3 PUT outputs to .proxies/ with ContentType + tags
      --> VastDB INSERT proxy_outputs (shared file_id with exr-inspector)
      --> Kafka publish proxy.generated event

Output Path Convention

Outputs are written to a .proxies/ sibling directory of the source file in the same bucket:

s3://bucket/renders/shot_010/beauty.0001.exr              (source)
s3://bucket/renders/shot_010/.proxies/beauty.0001_thumb.jpg (thumbnail)
s3://bucket/renders/shot_010/.proxies/beauty.0001_proxy.mp4 (proxy)

The .proxies/ prefix keeps outputs hidden from NFS directory listings (dotfile convention) while remaining accessible via S3 presigned URLs for the SpaceHarbor web application.

What It Generates

Output Format Resolution Color Space Content-Type Use Case
Thumbnail JPEG (quality 85) 256x256 sRGB image/jpeg Web UI previews, asset browsers
Proxy H.264 MP4 (CRF 23, faststart) 1920x1080 Rec709 video/mp4 Client review, editorial playback

Proxies are encoded with -movflags +faststart so browsers can begin streaming immediately via presigned URLs without downloading the full file.

Parallel Pipeline with exr-inspector

Both functions trigger on the same ElementCreated event and run in parallel:

ElementCreated (.exr suffix filter)
  |
  +---> exr-inspector        --> files, parts, channels, attributes tables
  |     (256KB Range GET)
  |
  +---> oiio-proxy-generator --> proxy_outputs table
        (full S3 GET)

Shared file_id computation (SHA256(path + mtime + MD5(path))[:16]) enables JOINs across all tables. The .exr suffix filter prevents infinite loops since outputs are .jpg and .mp4.

SpaceHarbor Integration

The SpaceHarbor MAM application serves thumbnails and proxies to browsers via S3 presigned URLs:

Browser -> Fastify backend (query VastDB for S3 keys)
        -> Generate presigned URL (5min for thumbs, 60min for proxies)
        -> Browser fetches directly from VAST S3

Quick Start

# Install dependencies
pip install -r functions/oiio_proxy_generator/requirements.txt

# Run tests
cd functions/oiio_proxy_generator
python -m pytest test_vast_db_persistence.py -v

# Build, fix, push, deploy
vastde functions build oiio-proxy-generator \
  --target functions/oiio_proxy_generator --pull-policy never
docker build --platform linux/amd64 --no-cache \
  -t $REGISTRY/oiio-proxy-generator:v2.0.0 -f Dockerfile.fix .
docker push $REGISTRY/oiio-proxy-generator:v2.0.0
vastde functions update oiio-proxy-generator --image-tag v2.0.0
vastde pipelines deploy $PIPELINE_NAME

Wiki Pages

Page Description
Architecture Event flow, module design, color pipeline, design decisions
Configuration Environment variables, resource sizing, pipeline setup
Database Schema proxy_outputs table, file_id computation, JOIN examples
Deployment Guide Build, push, create function, configure pipeline
Performance Optimization Resource allocation, autoscaling, bottlenecks
Troubleshooting Common issues and solutions

Requirements

  • VAST Cluster 5.4+ with DataEngine enabled
  • vastde CLI v5.4.1+
  • Docker with min-api-version: "1.38"
  • Python 3.12 (container runtime)
  • System tools: oiiotool (OpenImageIO 2.2+), ffmpeg
  • S3 bucket with DataEngine element trigger (suffix filter: .exr)
  • Database-enabled bucket for VAST DataBase persistence

Clone this wiki locally