DeepStream-Inspired Multiprocessing via DLPack & CUDA IPC for Zero-Copy Video Analytics

This repository demonstrates a high-performance, multi-process architecture for parallel multi-stream video decoding (NVDEC) and batched GPU inference (YOLO) in Python.

By leveraging DLPack and PyTorch CUDA IPC, this architecture effectively implements a pure-Python equivalent of NVIDIA DeepStream's nvstreammux component, bypassing the Python Global Interpreter Lock (GIL) and eliminating costly PCIe CPU-GPU memory transfers.

This work was originally documented and discussed in VALI Issue #183.

📊 Performance Discovery & Benchmarks

A critical phase of this research was determining whether the complexity of a Multiprocessing IPC architecture was justified over a more traditional Multithreaded approach.

During the technical exchange in VALI Issue #183, the library author, provided a highly optimized multithreaded implementation. This pattern releases the Python GIL within the native C++ decoding/conversion modules, theoretically allowing threads to run in parallel with minimal contention.

To find the true "Speed of Light" (SOL) for this pipeline, I conducted a head-to-head discovery benchmark:

The Hypothesis

While Pipeline B (Multithreading) is architecturally simpler and more VRAM-efficient (sharing a single CUDA context), I hypothesized that Pipeline A (Multiprocessing IPC) would still yield higher throughput. The goal was to prove that for high-density inference, total process isolation is required to fully saturate the i9-14900K’s execution units and the RTX 4060 Ti’s NVDEC/TensorCores.

Test Environment

• CPU: Intel Core i9-14900K (24-core, 32-thread)
• GPU: NVIDIA GeForce RTX 4060 Ti (Driver: 590.48.01, CUDA 12.x)
• RAM: 128 GiB DDR5 (4000 MT/s)
• OS: Ubuntu 24.04.4 LTS / Python 3.10
• Workload: 4x 1080p concurrent streams $\rightarrow$ NVDEC $\rightarrow$ Batched YOLO11s-Pose.

The Results

Metric	Pipeline A: MP + DLPack IPC (This Architecture)	Pipeline B: MT + C++ GIL Release (Author Pattern)	Delta
Total Throughput	395.6 FPS	272.0 FPS	+31.2%
Avg FPS per Stream	98.9 FPS	68.0 FPS	+31.2%
Architectural Cost	High (Multi-Context / IPC)	Low (Single Context)	-

The Discovery: +31.2% Hardware Saturation

The data revealed a significant finding: Even with the GIL released in native code, the Python-level coordination for batch assembly in a multithreaded environment creates enough overhead to leave 31.2% of the GPU's potential on the table.

By switching to Multiprocessing with DLPack IPC, I bypassed the remaining Python bottlenecks entirely. Each decoder process operates in its own sandbox, feeding a "Predictor" process that functions as a high-speed GPU-memory aggregator—essentially a pure-software implementation of a hardware stream-multiplexer.

🧠 The Architecture: DeepStream nvstreammux in pure Python

When building scalable video analytics pipelines in Python, developers usually hit the GIL bottleneck. Multi-threading falls short for high-throughput RTSP streams because Python struggles to coordinate concurrent GPU decode and inference loops, even when native C++ extensions release the GIL.

NVIDIA DeepStream solves this elegantly using nvstreammux, which collects async video buffers directly in GPU memory, waits for a batch to form, and pushes the batched pointers to nvinfer.

To replicate this zero-copy, highly concurrent behavior in Python, I designed an N-to-1 Producer-Consumer Multiprocessing Architecture:

1. N Decoder Processes (Producers): Each process owns an independent CUDA context. It uses VALI to trigger hardware decoding (NVDEC) from RTSP/Disk directly into GPU memory.
2. CUDA IPC & DLPack (The Bridge): Instead of pickling pointers (which fails) or transferring frames to the CPU (which incurs severe PCIe latency), decoders inject frames into pre-allocated PyTorch tensors via DLPack. These tensors expose CUDA IPC handles via tensor.share_memory_(), which are safely passed across a standard multiprocessing.Queue.
3. 1 Predictor Process (Consumer): A centralized process aggregates these cross-process GPU memory handles, continuously forming batches (like nvstreammux), and feeds them to YOLO for maximized TensorCore utilization.

⚙️ The DLPack "Reverse-Mapping" Trick

The secret to this architecture is reversing the standard DLPack flow. Instead of converting a decoded library surface to a tensor, we pre-allocate a PyTorch CUDA tensor, expose it via DLPack to the C++ decoding API, and decode/color-convert directly into the PyTorch memory space.

# 1. Pre-allocate packed RGB PyTorch CUDA tensor
tensor_rgb = torch.empty((target_h, target_w * 3), device="cuda", dtype=torch.uint8)

# 2. Wrap via DLPack into a VALI Surface (Zero-Copy View)
dlpack_capsule = torch.utils.dlpack.to_dlpack(tensor_rgb)
surf_rgb = vali.Surface.from_dlpack(dlpack_capsule, vali.PixelFormat.RGB)

# 3. Hardware Decode (NV12) -> Convert directly into our Tensor's memory
py_ud.RunAsync(surf_nv12, surf_rgb)
torch.cuda.synchronize()

# 4. Expose CUDA IPC handle and send across multiprocessing boundary
tensor_rgb.share_memory_()
decode_queue.put(tensor_rgb)

🚀 Running the Benchmark

Installation

Ensure you have an NVIDIA GPU, CUDA toolkit installed, and run:

pip install -r requirements.txt

Execution

Run the comprehensive benchmark script to compare both pipelines on your own hardware:

python benchmark_para_flow_architecture.py \
    --source path/to/video.mp4 \
    --model yolo11s-pose.pt \
    --streams 4 \
    --batch-size 4 \
    --duration 30 \
    --gpu 0

Detailed logging and NVTX markers are included in the code, which can be used with NVIDIA Nsight Systems to visually verify concurrency and CUDA IPC handshakes:

nsys profile --trace=cuda,nvtx,osrt python benchmark_para_flow_architecture.py --source path/to/video.mp4

🤝 Acknowledgments

Special thanks to Roman Arzumanyan, the creator of the phenomenal VALI library, for his guidance, multithreading samples, and deep technical discussions in Issue #183.