vlm-token-budget-lab

A small simulation toolkit for studying visual-token budget strategies in Vision-Language Model (VLM) inference.

Why this project

Recent VLM work emphasizes efficiency: prune or compress visual tokens to cut latency while preserving answer quality. This repo provides a reproducible sandbox for that tradeoff analysis without requiring GPU-heavy models.

What it does

• Simulates layer-wise visual token saliency.
• Applies pruning schedules (once or progressive).
• Estimates compute cost reduction using a simple attention-cost proxy.
• Reports a quality-retention proxy from preserved token saliency.
• Exports layer statistics to CSV.

Install

pip install -e .

Quickstart

vlm-budget simulate \
  --num-layers 32 \
  --image-tokens 576 \
  --text-tokens 128 \
  --start-layer 2 \
  --prune-ratio 0.5 \
  --mode once \
  --seed 42

Example output:

baseline_cost=15859712
pruned_cost=9295872
speedup=1.7061x
cost_reduction=41.3864%
quality_retention=0.8612

Export per-layer stats:

vlm-budget simulate --save-csv outputs/layer_stats.csv

Notes

• quality_retention is a proxy, not benchmark accuracy.
• Use this tool for planning and ablation design before full-model experiments.

Acknowledgement

Efficiency motivation inspired by open VLM acceleration work, including:

• https://github.com/pkunlp-icler/FastV

This implementation is original and intentionally lightweight.

License

MIT