CAMAL Open-Source Release

This repository contains the code release for CAMAL.

What is included

• amazoncat13k/
• oneshotwikilinks/
• wiki10/
• synthetic_base/
• synthetic_tricky/
• synthetic_complex/
• requirements.txt
• LICENSE

In each dataset folder, we provide the runnable Python files needed for:

• corral_cost.py: CAMAL / our method
• llm_agent.py: standalone LLM agent code
• llm_only.py: standalone LLM baseline
• bandit.py: traditional bandit baseline when a standalone script exists
• make_data.py: preprocessing

We do not include notebooks, cached files, local results, figures, archived experiments, or large raw datasets in this release.

Core Method

CAMAL combines two main components:

• Cost-aware sampling: the master policy selects between a low-cost bandit expert and a higher-cost LLM expert using a dynamic penalty based on reward/cost trade-offs.
• History-conditioned LLM agent: the LLM prompt includes recent interaction history so the model can adapt online without gradient updates.

Folder Layout

camal_release/
  amazoncat13k/
  oneshotwikilinks/
  wiki10/
  synthetic_base/
  synthetic_tricky/
  synthetic_complex/
  requirements.txt
  LICENSE

Naming Convention

For consistency across datasets:

• our method is named corral_cost.py
• the standalone LLM agent is named llm_agent.py
• the standalone LLM baseline is named llm_only.py
• the traditional method is named bandit.py

Some source files had different names in our research workspace. In this release, we rename them to keep the interface consistent across datasets.

Data Preparation

We include the small synthetic datasets directly in this repository:

• synthetic_base/weather_bandit_100.json
• synthetic_tricky/weather_bandit_2000_storm_like.json
• synthetic_complex/weather_bandit_2000.json

For the large real-world datasets, please download them separately, place them inside the corresponding dataset folder, and run make_data.py from that folder.

AmazonCat-13K

Download link: AmazonCat-13K at Extreme Classification Repository

Expected local files under amazoncat13k/:

• AmazonCat-13K/Yf.txt
• AmazonCat-13K/trn.json.gz

Preprocess:

cd amazoncat13k
python make_data.py

OneShotWikiLinks-311

Download link: OneShotWikiLinks on Kaggle

Expected local file under oneshotwikilinks/:

• shuffled_dedup_entities.tsv

Preprocess:

cd oneshotwikilinks
cut -f1 shuffled_dedup_entities.tsv | sort -S50% | uniq -c | sort -S10% -k1rn > entityfreq
python make_data.py

Note: on Windows, generate entityfreq with an equivalent local command or inside WSL/Git Bash.

Wiki10-642

Download link: Wiki10+ dataset

Expected local files under wiki10/Wiki10-31K/:

• label_map.txt
• train_raw_texts.txt
• Y.trn.npz

Preprocess:

cd wiki10
python make_data.py

Synthetic-Base

Included file under synthetic_base/:

• weather_bandit_100.json

Preprocess:

cd synthetic_base
python make_data.py

Synthetic-Tricky

Included file under synthetic_tricky/:

• weather_bandit_2000_storm_like.json

Preprocess:

cd synthetic_tricky
python make_data.py

Synthetic-Complex

Included file under synthetic_complex/:

• weather_bandit_2000.json

Preprocess:

cd synthetic_complex
python make_data.py

Running Experiments

Run the commands below from the dataset folder after preprocessing and after setting the desired local LLM checkpoint path in --llm-type or --llm_type, depending on the script.

Our Method

python corral_cost.py --llm-type /path/to/Qwen2.5-1.5B-Instruct-GPTQ-Int4 --output-path ./results.json

Traditional Bandit

If the dataset folder contains bandit.py:

python bandit.py

Standalone LLM

python llm_only.py

LLM Agent

python llm_agent.py

Some llm_only.py and llm_agent.py scripts use --llm_type instead of --llm-type. Please check the CLI arguments in the corresponding file before running.

Results Format

The main output JSON files contain the paper's reward/cost related statistics. Important fields include:

• num_llms: number of rounds routed to the LLM policy
• avreward_corral: running reward of CAMAL
• avreward_bandit: running reward of the low-cost bandit
• total_llm_time: accumulated LLM runtime
• total_bandit_time: accumulated bandit runtime
• alpha_t: dynamic penalty trace when recorded
• master_p_llm: master probability assigned to the LLM when recorded

LLM usage (%) in the paper is computed from num_llms divided by the number of evaluated examples or rounds.

Scope Of This Release

This release is intentionally compact.

• Full CAMAL is included as corral_cost.py.
• Standalone LLM agent code is included as llm_agent.py.
• Standalone LLM baselines are included as llm_only.py.
• Traditional baselines are reduced to bandit.py only and are now present in all six dataset folders.
• We do not include additional legacy routing variants, notebook experiments, plotting scripts, or archived versions.
• synthetic_base and synthetic_tricky reuse the same Python CAMAL runner structure as the complex synthetic setup, because in our original research workspace these two smaller synthetic settings only had notebook variants.
• For synthetic datasets, bandit.py, llm_only.py, and llm_agent.py are provided here as runnable .py files derived from our original notebook implementations.

Ablation Notes

• w/o cost penalty: use corral_cost.py with a fixed penalty such as --alpha 0.
• w/o history: in this release, we do not duplicate additional no-history runners only for packaging purposes. The included corral_cost.py is the full history-enabled version.

Environment Notes

• Our original experiments were developed in a Linux GPU environment.
• Some scripts assume relative paths and local model checkpoints.
• Some preprocessing commands shown above are Unix-style. Use equivalent commands on Windows if needed.