Local AI Compute Broker on Slurm

This repository now has two layers:

• a legacy set of Slurm and local-model helper scripts
• an in-progress local AI compute broker for MCP-capable agents under broker/

The broker direction is the primary project trajectory:

• frontier remote models remain the orchestrators
• local cluster jobs perform token-heavy analysis
• only compact structured outputs should flow back to the remote model by default

Start here for the broker project:

• docs/README.md
• docs/quickstart.md
• docs/architecture.md
• docs/rag-compression.md
• docs/mcp-tools.md
• docs/roadmap.md

Smoke validation:

• bash tests/e2e/run_smoke_suite.sh
• bash tests/e2e/run_smoke_suite.sh --with-loopback-bind for the fake-llama.cpp RAG runtime path

Verified MCP client status:

• Codex CLI can call the broker MCP tools and submit real Slurm-backed rag_compress jobs.
• GitHub Copilot CLI wiring is still not verified end-to-end and should be treated as experimental.

Opt-in Codex broker profiles:

• examples/mcp-clients/install_codex_profiles.sh installs two profile-scoped Codex configs under ~/.codex/.
• codex -p slurm-broker enables the Slurm-backed broker MCP server for that session only.
• codex -p local-broker enables the local-command broker MCP server for that session only.
• Plain codex startup remains unchanged unless you explicitly select one of those profiles.

Legacy llama.cpp workflow:

Scripts to run llama.cpp server on SLURM and connect Claude CLI to it remain in this repository while the broker architecture is being built.

Features:

• Automated server submission to SLURM with optimized settings
• Central registry for discovering servers without shared filesystem access
• Pre-configured model profiles for common LLMs
• Automatic cleanup and health monitoring
• Email notifications when servers are ready
• Push notifications via ntfy.sh

Quick Start

For Users with Shared Filesystem Access

1. Submit llama.cpp server with a saved configuration:

   deploy/slurm/submit_llama.sh --config qwen3-30b

2. Wait for server to start (check logs):

   tail -f llama_server_<job_id>.log

3. Connect Claude CLI to the server:

   deploy/slurm/connect_claude_llama.sh <job_id>

For Users Without Shared Filesystem Access

1. List available servers from the registry:

   curl http://your-registry-server:5000/servers | jq

2. Create SSH tunnel to the server:

   ssh -L 8080:<host>:<port> your-login-node

3. Connect Claude to the tunneled server:

   export ANTHROPIC_BASE_URL="http://localhost:8080"
   claude

See REGISTRY_SETUP.md for registry server setup.

Usage

Submit llama.cpp Server

deploy/slurm/submit_llama.sh [OPTIONS]

Options:

• --config NAME - Use predefined model config
• --list-configs - List available model configurations
• --model FILE - Path to GGUF model file
• --time TIME - Time limit (default: 8:00:00)
• --no-time-limit - Remove time limit entirely
• --partition PART - SLURM partition (default: none)
• --qos QOS - SLURM QOS (default: none)
• --cpus N - Number of CPUs (default: 8)
• --mem SIZE - Memory allocation (default: 32G)
• --gpus N - Number of GPUs (default: 1)
• --context N - Context size (default: 131072)
• --gpu-layers N - GPU layers, -1 for all (default: -1)
• --extra-args STR - Additional llama-server arguments
• --email EMAIL - Email address for notification when server is ready
• --ntfy-topic TOPIC - Ntfy topic for push notifications (default: llama-servers)
• --ntfy-server URL - Ntfy server URL (default: https://ntfy.sh)
• --help, -h - Show help message

Examples:

# Use saved configuration
deploy/slurm/submit_llama.sh --config qwen3-30b

# List available configurations
deploy/slurm/submit_llama.sh --list-configs

# No time limit with saved config
deploy/slurm/submit_llama.sh --config glm-4.7 --no-time-limit

# With email notification
deploy/slurm/submit_llama.sh --config qwen3-30b --email user@example.com

# With ntfy push notification
deploy/slurm/submit_llama.sh --config qwen3-30b --ntfy-topic my-llama-servers

# Custom model file
deploy/slurm/submit_llama.sh --model ~/.cache/llama.cpp/model.gguf

# Custom settings
deploy/slurm/submit_llama.sh --model model.gguf --gpus 2 --context 16384

Model Configurations

Model configurations are stored in model_configs/*.conf. Each config sets optimal parameters based on claude-code-tools recommendations.

Server settings (all models):

• Context: 131072 tokens (128K)
• Batch size: 32768
• Ubatch: 1024
• Parallel slots: 1
• Jinja templating: enabled

Create a new configuration:

cat > model_configs/my-model.conf <<EOF
MODEL_FILE="~/.cache/llama.cpp/my-model.gguf"
CPUS=16
MEM="64G"
GPUS=2
CONTEXT_SIZE=131072
N_GPU_LAYERS=-1
EXTRA_ARGS=""
EOF

Available configurations:

Config	Model	GPUs	Memory	GPU Requirements	Notes
gpt-oss-20b.p40	GPT-OSS 20B MXFP4	1	32G	1x P40 24GB	64K context, smaller/faster single-GPU P40 profile
qwen3-30b	Qwen3 30B Q4_K_M	2	64G	2x A100 80GB or 4x V100 32GB	~18GB model + 13GB KV cache/GPU
qwen3-30b.p40	Qwen3 30B Q4_K_M	1	32G	1x P40 24GB	16K context, reduced batch for 24GB cards
qwen3-coder-30b	Qwen3 Coder 30B Q8_0	2	64G	2x A100 80GB or 4x V100 32GB	~32GB model + 13GB KV cache/GPU
qwen3-80b	Qwen3 80B Q4_K_XL	4	128G	4x A100 80GB only	~47GB model + 13GB KV cache/GPU
glm-4.7	GLM-4.7 Flash Q4_K_M	2	32G	2x A100 80GB only	~16GB model + 13GB KV cache/GPU, DeepSeek2 gating

GPU VRAM Notes:

• A100 80GB: Can run all configurations
• V100 32GB: Can run qwen3-30b and qwen3-coder-30b with 4 GPUs, NOT glm-4.7 or qwen3-80b
• V100 16GB: Not recommended for these models with 128K context
• P40 24GB: Best fits here are gpt-oss-20b.p40 for a smaller/faster 64K profile and qwen3-30b.p40 for a larger 16K profile
• KV cache at 128K context requires ~13GB VRAM per GPU for all models

Connect Claude CLI

deploy/slurm/connect_claude_llama.sh <job_id> [working_directory]

Run Codex End To End

deploy/slurm/run_codex_llama.sh \
  --config gpt-oss-20b.p40 \
  --partition hpc \
  --prompt "Reply with exactly: P40_OK"

This flow:

• submits the llama.cpp Slurm job
• waits for the server connection file
• starts the supported Codex compatibility proxy
• runs codex exec against the local /v1/responses endpoint
• cancels the Slurm job on exit unless --keep-job is set

The repo-root run_codex_llama.sh remains as a compatibility shim. The repo-root submit_llama.sh and connect_claude_llama.sh also remain as compatibility shims.

Arguments:

• job_id - SLURM job ID of the running llama.cpp server
• working_directory - Optional directory where Claude should run (default: current directory)

Examples:

# Connect from current directory
deploy/slurm/connect_claude_llama.sh 2883398

# Connect and work in a specific directory
deploy/slurm/connect_claude_llama.sh 2883398 /path/to/my/project

# Pass additional arguments to Claude
deploy/slurm/connect_claude_llama.sh 2883398 ~/code --resume abc123

Note: On first connection to a directory, Claude will ask you to confirm trust of the workspace. Make sure to run connect_claude_llama.sh from an interactive terminal (not a script). If you see the "trust this folder" prompt, select "Yes" to continue. You can also pre-approve with: claude trust /path/to/workspace

Known Issue: Some directories may cause Claude to hang indefinitely when connecting to local llama.cpp servers, even though they work fine with Anthropic's API. This appears to be a Claude Code bug with certain directory configurations. If you experience this, try:

1. Use a different working directory that doesn't have the issue
2. Start Claude in a working directory, then access problematic directory files as needed
3. Use Claude with the official Anthropic API for those directories

Lists active llama.cpp server jobs if no job ID is provided.

Manual Connection

If you prefer to connect manually:

source llama_server_connection_<job_id>.txt
source setup_claude_env.sh
claude

Managing Jobs

Check job status:

squeue -u $USER -n llama-server

List available servers (via registry):

./list_servers.sh
./list_servers.sh --owner $USER

View logs:

tail -f llama_server_<job_id>.log

Cancel job:

scancel <job_id>

Server Discovery & Registry

NEW: Central registry service for discovering servers without shared filesystem access.

To enable, set the registry URL:

export REGISTRY_URL="http://your-registry-server:5000"
deploy/slurm/submit_llama.sh --config qwen3-30b

Servers will auto-register on startup and unregister on shutdown. See REGISTRY_SETUP.md for full setup guide.

Email Notifications

Get notified by email when your server is ready:

# Specify email when submitting
deploy/slurm/submit_llama.sh --config qwen3-30b --email user@example.com

# Or set default email in your environment
export NOTIFY_EMAIL="user@example.com"
deploy/slurm/submit_llama.sh --config qwen3-30b

The email includes:

• Server connection details (host, port, job ID)
• Multiple connection methods (direct, SSH tunnel, web dashboard)
• Commands to connect Claude
• Monitoring and management commands

Requirements:

• System must have mail, sendmail, or mutt installed
• SMTP service configured on compute nodes or login nodes

Push Notifications (ntfy)

Get instant push notifications to your phone or desktop when servers are ready using ntfy.sh:

# Use default public ntfy.sh server
deploy/slurm/submit_llama.sh --config qwen3-30b --ntfy-topic my-llama-servers

# Use custom ntfy server
deploy/slurm/submit_llama.sh --config qwen3-30b --ntfy-topic servers --ntfy-server https://ntfy.mycompany.com

# Set default topic in environment
export NTFY_TOPIC="my-llama-servers"
deploy/slurm/submit_llama.sh --config qwen3-30b

To receive notifications:

1. Install ntfy app on your phone (iOS, Android)
2. Subscribe to your topic (e.g., "my-llama-servers")
3. Submit jobs with --ntfy-topic

Or on desktop:

# Subscribe via CLI
ntfy subscribe my-llama-servers

# Or via web
open https://ntfy.sh/my-llama-servers

Features:

• Instant push notifications to phone/desktop
• No authentication required (for public ntfy.sh)
• Includes clickable link to web dashboard
• Works from anywhere (no VPN needed)
• Multiple devices can subscribe to same topic

Privacy note: Public ntfy.sh topics are visible to anyone who knows the topic name. Use unique topic names or deploy your own ntfy server for privacy.

Files

• deploy/slurm/submit_llama.sh - Main script to submit llama.cpp server to SLURM
• deploy/slurm/llama_server.slurm - SLURM batch script that runs llama.cpp server
• deploy/slurm/connect_claude_llama.sh - Connect Claude CLI to running llama.cpp server
• setup_claude_env.sh - Environment configuration for Claude CLI
• model_configs/*.conf - Model configuration files
• llama_server_connection_<job_id>.txt - Auto-generated connection info
• registry_server.py - Central registry service
• register_server.sh - Auto-register servers with registry
• list_servers.sh - List available servers from registry
• send_notification.sh - Email notification when server is ready
• send_ntfy_notification.sh - Push notification via ntfy
• REGISTRY_SETUP.md - Registry setup and usage guide

The root-level submit_llama.sh, submit_ollama.sh, connect_claude_llama.sh, llama_server.slurm, and ollama_server.slurm remain as compatibility shims.

Requirements

• llama.cpp with server support (llama-server binary)
  • Important: Must be compiled with CUDA support for your GPU architecture
  • For A100 GPUs (compute capability 8.0): compile with GGML_CUDA_COMPUTE_CAP=80
  • For V100 GPUs (compute capability 7.0): compile with GGML_CUDA_COMPUTE_CAP=70
  • See llama.cpp compilation guide
• GGUF model files in ~/.cache/llama.cpp/
• Claude CLI installed
• Python 3 (for port selection)
• curl (for health checks)
• NVIDIA GPUs with CUDA support
  • A100 80GB: Recommended, supports all model configurations
  • V100 32GB: Supports qwen3-30b and qwen3-coder-30b (with 4 GPUs)
  • V100 16GB: Not recommended for 128K context models

Troubleshooting

CUDA error: "no kernel image is available for execution":

• Your llama-server was compiled for a different GPU architecture

• Check your GPU compute capability: nvidia-smi --query-gpu=compute_cap --format=csv

• Recompile llama.cpp with correct architecture:

  cd ~/path/to/llama.cpp
  make clean
  GGML_CUDA=1 GGML_CUDA_COMPUTE_CAP=80 make llama-server  # For A100 (8.0)

• Or use pre-built binaries matching your GPU

Server won't start:

• Check logs: cat llama_server_<job_id>.log
• Verify llama-server is installed: which llama-server
• Check node resources: squeue -j <job_id>
• Verify model file exists: ls -lh ~/.cache/llama.cpp/

Connection issues:

• Ensure connection file exists: ls llama_server_connection_*.txt
• Check server is running: squeue -j <job_id>
• Verify network access between nodes

Model issues:

• Check available models: ls ~/.cache/llama.cpp/*.gguf
• Verify model path in configuration
• Try with different quantization (Q4_K_M, Q8_0, etc.)

Out of memory errors:

• Check GPU VRAM: nvidia-smi
• GLM-4.7 and qwen3-80b require A100 80GB GPUs
• qwen3-30b models can use V100 32GB with 4 GPUs
• Reduce context size: --context 65536 (but may impact Claude Code performance)
• Use lower quantization: Q4_K_M instead of Q8_0