Skip to content

Commit 251c296

Browse files
himmi-01himmi-01
committed
feat: regression guard, agent card, recommend command, external datasets
- feat(1B): Regression Guard — auto-detect score drops after every run-benchmark; new 'evalmonkey guard' command exits with code 1 for CI/CD gating. Configurable via EVAL_REGRESSION_THRESHOLD env var (default: 5 pts). - feat(1C): Agent Card — 'evalmonkey report' generates shareable Markdown with shields.io badge, per-scenario score table, and production reliability metric. Paste the badge directly into your README. - feat(1A): 'evalmonkey recommend' — reads agent_type from evalmonkey.yaml and shows curated benchmark suite (research_agent, coding_agent, rag_agent, customer_support, voice_agent, safety_agent, general). - feat(phase2): External and Private Datasets - '--dataset my_cases.jsonl' flag for local JSONL/JSON/CSV files - 'hf::org/dataset' prefix to load any HuggingFace dataset directly - 'confident-ai::dataset_id' / 'braintrust::ref' / 'langsmith::id' prefixes to use EvalMonkey as chaos+scoring harness on top of your existing eval platform datasets (API key from .env) - Generic REST harness via 'private_benchmarks' block in evalmonkey.yaml - docs: add sample agent benchmark cards to assets/agent_cards/ - tests: 53 new unit tests (all mocked, no network), 149 total passing
1 parent 788fcfd commit 251c296

17 files changed

Lines changed: 2285 additions & 4 deletions

‎.env.example‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,3 +69,32 @@ LANGFUSE_SECRET_KEY="sk-lf-..."
6969
# Used by: evalmonkey generate-evals --langfuse-dataset <name>
7070
# demo_rag_app.sh (automatic if keys are set)
7171
# LANGFUSE_DATASET="evalmonkey_failures"
72+
73+
# ----------------------------------------
74+
# 5. Regression Guard (Optional)
75+
# ----------------------------------------
76+
# Score drop (in points) that triggers a regression warning after run-benchmark
77+
# and causes `evalmonkey guard` to exit with code 1.
78+
# Default: 5 (i.e. a drop of 5+ points vs the previous baseline is flagged)
79+
EVAL_REGRESSION_THRESHOLD=5
80+
81+
# ----------------------------------------
82+
# 6. External Dataset Providers (Optional)
83+
# ----------------------------------------
84+
# Use EvalMonkey as the chaos + scoring harness on top of datasets you already
85+
# maintain in eval platforms you subscribe to.
86+
#
87+
# Confident AI (DeepEval cloud)
88+
# Get your key: https://app.confident-ai.com → Settings → API Keys
89+
# Usage: evalmonkey run-benchmark --scenario confident-ai::<dataset_id>
90+
CONFIDENT_AI_API_KEY="conf-..."
91+
92+
# Braintrust
93+
# Get your key: https://www.braintrustdata.com → Settings → API Keys
94+
# Usage: evalmonkey run-benchmark --scenario braintrust::<project>/<dataset>
95+
BRAINTRUST_API_KEY="bt-..."
96+
97+
# LangSmith (LangChain)
98+
# Get your key: https://smith.langchain.com → Settings → API Keys
99+
# Usage: evalmonkey run-benchmark --scenario langsmith::<dataset_id>
100+
LANGSMITH_API_KEY="ls__..."

‎assets/agent_cards/README.md‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# EvalMonkey Agent Cards
2+
3+
Sample benchmark report cards generated by `evalmonkey report`.
4+
5+
## EvalMonkey Sample Apps
6+
7+
- [EvalMonkey RAG App](./rag_app.md) — RAG Agent (Demo)
8+
- [EvalMonkey Coding Agent](./coding_agent.md) — Coding Agent (Demo)
9+
10+
## Open-Source Agents (from the EvalMonkey Leaderboard)
11+
12+
- [GPT Researcher](./gpt_researcher.md) — Rank #1, Production Reliability: **57**
13+
- [OpenResearcher](./openresearcher.md) — Rank #2, Production Reliability: **55**
14+
15+
---
16+
17+
Generate your own card:
18+
```bash
19+
evalmonkey report --output my_agent_card.md
20+
```

‎assets/agent_cards/coding_agent.md‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# Agent Benchmark Card — EvalMonkey Coding Agent
2+
3+
[![EvalMonkey Score](https://img.shields.io/badge/EvalMonkey-Score%3A73-yellow)](https://github.com/Corbell-AI/evalmonkey)
4+
[![Production Reliability](https://img.shields.io/badge/Production%20Reliability-Score%3A67-yellow)](https://github.com/Corbell-AI/evalmonkey)
5+
6+
> Evaluated by [EvalMonkey](https://github.com/Corbell-AI/evalmonkey) · 2026-05-28
7+
8+
## Overview
9+
10+
| Field | Value |
11+
|-------|-------|
12+
| Agent | [EvalMonkey Coding Agent](https://github.com/Corbell-AI/evalmonkey) |
13+
| Type | Coding Agent (Demo) |
14+
| Framework | LiteLLM + FastAPI |
15+
| Agent Type Config | `agent_type: coding_agent` |
16+
| Eval Judge | gpt-4o |
17+
| Chaos Profiles Tested | 4 (`code_syntax_break`, `code_wrong_language`, `code_context_strip`, `client_prompt_injection`) |
18+
19+
## Scores
20+
21+
| Benchmark | Baseline | Chaos | Production Reliability |
22+
|-----------|:--------:|:-----:|:----------------------:|
23+
| `human-eval` | 🟢 **78** | 62 | 71.6 |
24+
| `mbpp` | 🟢 **82** | 68 | 76.4 |
25+
| `apps` | 🟡 **59** | 44 | 53.0 |
26+
27+
| **Overall** | **73** | **58** | **67.0** |
28+
29+
## Production Reliability
30+
31+
```
32+
Production Reliability = (baseline × 0.6) + (chaos × 0.4)
33+
= (73 × 0.6) + (58 × 0.4)
34+
= 67.0
35+
```
36+
37+
## Analysis
38+
39+
> EvalMonkey's built-in coding demo. Strong on basic Python, weaker on competitive challenges.
40+
41+
## Reproduce This Benchmark
42+
43+
```bash
44+
# Clone EvalMonkey
45+
git clone https://github.com/Corbell-AI/evalmonkey.git
46+
cd evalmonkey && pip install -e .
47+
48+
# Set up your .env
49+
cp .env.example .env # Add your OPENAI_API_KEY or EVAL_MODEL
50+
51+
# Run the Coding Agent (Demo) sample app
52+
python apps/coding_agent/app.py &
53+
54+
evalmonkey run-benchmark --scenario human-eval --sample-agent coding_agent
55+
evalmonkey run-benchmark --scenario mbpp --sample-agent coding_agent
56+
evalmonkey run-benchmark --scenario apps --sample-agent coding_agent
57+
58+
# Chaos test
59+
evalmonkey run-chaos --scenario human-eval --chaos-profile code_syntax_break --sample-agent coding_agent
60+
61+
# Generate this card
62+
evalmonkey report --output agent_card.md
63+
```
64+
65+
## Embed This Badge
66+
67+
```markdown
68+
[![EvalMonkey Score](https://img.shields.io/badge/EvalMonkey-Score%3A73-yellow)](https://github.com/Corbell-AI/evalmonkey)
69+
```
70+
71+
---
72+
73+
*Generated by [EvalMonkey](https://github.com/Corbell-AI/evalmonkey) — the open-source agent benchmarking and chaos framework.*

‎assets/agent_cards/gpt_researcher.md‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# Agent Benchmark Card — GPT Researcher
2+
3+
[![Production Reliability](https://img.shields.io/badge/Production%20Reliability-Score%3A57-orange)](https://github.com/assafelovic/gpt-researcher)
4+
[![GitHub](https://img.shields.io/badge/GitHub-View%20Repo-181717?logo=github)](https://github.com/assafelovic/gpt-researcher)
5+
6+
> Evaluated by [EvalMonkey](https://github.com/Corbell-AI/evalmonkey) · 2026-05-28
7+
8+
## Overview
9+
10+
| Field | Value |
11+
|-------|-------|
12+
| Agent | [GPT Researcher](https://github.com/assafelovic/gpt-researcher) |
13+
| Type | Deep Research Agent |
14+
| EvalMonkey Rank | 🏅 #1 of 10 open-source agents |
15+
| Eval Judge | Claude Sonnet 4.5 (AWS Bedrock) |
16+
| Chaos Profiles | `client_prompt_injection`, `client_schema_mutation` |
17+
18+
## Scores
19+
20+
| Benchmark | Baseline | Chaos | Production Reliability |
21+
|-----------|:--------:|:-----:|:----------------------:|
22+
| `hotpotqa` | 🟢 **66** | 17 | 46.4 |
23+
| `truthfulqa` | 🟢 **65** | 48 | 58.2 |
24+
| `mmlu` | 🟡 **56** | 16 | 40.0 |
25+
26+
| **Overall** | **66** | **43** | **57** |
27+
28+
## Production Reliability
29+
30+
```
31+
Production Reliability = (baseline × 0.6) + (chaos × 0.4)
32+
= (66 × 0.6) + (43 × 0.4)
33+
= 57
34+
```
35+
36+
> Production Reliability measures how your agent performs under **real-world conditions** —
37+
> not just clean benchmark inputs, but also adversarial mutations like prompt injection,
38+
> schema corruption, and typo flooding.
39+
40+
## Analysis
41+
42+
> Highest baseline scorer. Dropped 23 pts under chaos — sensitive to prompt injection.
43+
44+
## How to Re-run This Benchmark
45+
46+
```bash
47+
# Install EvalMonkey
48+
pip install git+https://github.com/Corbell-AI/evalmonkey.git
49+
50+
# Start GPT Researcher on port 8000 (see its own README)
51+
52+
# Run the same benchmarks
53+
evalmonkey run-benchmark --scenario hotpotqa --target-url http://localhost:8000/solve
54+
evalmonkey run-benchmark --scenario truthfulqa --target-url http://localhost:8000/solve
55+
evalmonkey run-benchmark --scenario mmlu --target-url http://localhost:8000/solve
56+
57+
# Chaos test
58+
evalmonkey run-chaos --scenario hotpotqa --chaos-profile client_prompt_injection --target-url http://localhost:8000/solve
59+
60+
# Generate this card
61+
evalmonkey report --output agent_card.md
62+
```
63+
64+
---
65+
66+
*Generated by [EvalMonkey](https://github.com/Corbell-AI/evalmonkey) — the open-source agent benchmarking and chaos framework.*

‎assets/agent_cards/openresearcher.md‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# Agent Benchmark Card — OpenResearcher
2+
3+
[![Production Reliability](https://img.shields.io/badge/Production%20Reliability-Score%3A55-orange)](https://github.com/GAIR-NLP/OpenResearcher)
4+
[![GitHub](https://img.shields.io/badge/GitHub-View%20Repo-181717?logo=github)](https://github.com/GAIR-NLP/OpenResearcher)
5+
6+
> Evaluated by [EvalMonkey](https://github.com/Corbell-AI/evalmonkey) · 2026-05-28
7+
8+
## Overview
9+
10+
| Field | Value |
11+
|-------|-------|
12+
| Agent | [OpenResearcher](https://github.com/GAIR-NLP/OpenResearcher) |
13+
| Type | Scientific Research Agent |
14+
| EvalMonkey Rank | 🏅 #2 of 10 open-source agents |
15+
| Eval Judge | Claude Sonnet 4.5 (AWS Bedrock) |
16+
| Chaos Profiles | `client_prompt_injection`, `client_schema_mutation` |
17+
18+
## Scores
19+
20+
| Benchmark | Baseline | Chaos | Production Reliability |
21+
|-----------|:--------:|:-----:|:----------------------:|
22+
| `hotpotqa` | 🟢 **64** | 19 | 46.0 |
23+
| `truthfulqa` | 🟢 **63** | 47 | 56.6 |
24+
| `mmlu` | 🟡 **55** | 18 | 40.2 |
25+
26+
| **Overall** | **64** | **42** | **55** |
27+
28+
## Production Reliability
29+
30+
```
31+
Production Reliability = (baseline × 0.6) + (chaos × 0.4)
32+
= (64 × 0.6) + (42 × 0.4)
33+
= 55
34+
```
35+
36+
> Production Reliability measures how your agent performs under **real-world conditions** —
37+
> not just clean benchmark inputs, but also adversarial mutations like prompt injection,
38+
> schema corruption, and typo flooding.
39+
40+
## Analysis
41+
42+
> Strong research synthesis. Stable under schema mutation, weaker under prompt injection.
43+
44+
## How to Re-run This Benchmark
45+
46+
```bash
47+
# Install EvalMonkey
48+
pip install git+https://github.com/Corbell-AI/evalmonkey.git
49+
50+
# Start OpenResearcher on port 8000 (see its own README)
51+
52+
# Run the same benchmarks
53+
evalmonkey run-benchmark --scenario hotpotqa --target-url http://localhost:8000/solve
54+
evalmonkey run-benchmark --scenario truthfulqa --target-url http://localhost:8000/solve
55+
evalmonkey run-benchmark --scenario mmlu --target-url http://localhost:8000/solve
56+
57+
# Chaos test
58+
evalmonkey run-chaos --scenario hotpotqa --chaos-profile client_prompt_injection --target-url http://localhost:8000/solve
59+
60+
# Generate this card
61+
evalmonkey report --output agent_card.md
62+
```
63+
64+
---
65+
66+
*Generated by [EvalMonkey](https://github.com/Corbell-AI/evalmonkey) — the open-source agent benchmarking and chaos framework.*

‎assets/agent_cards/rag_app.md‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# Agent Benchmark Card — EvalMonkey RAG App
2+
3+
[![EvalMonkey Score](https://img.shields.io/badge/EvalMonkey-Score%3A71-yellow)](https://github.com/Corbell-AI/evalmonkey)
4+
[![Production Reliability](https://img.shields.io/badge/Production%20Reliability-Score%3A65-yellow)](https://github.com/Corbell-AI/evalmonkey)
5+
6+
> Evaluated by [EvalMonkey](https://github.com/Corbell-AI/evalmonkey) · 2026-05-28
7+
8+
## Overview
9+
10+
| Field | Value |
11+
|-------|-------|
12+
| Agent | [EvalMonkey RAG App](https://github.com/Corbell-AI/evalmonkey) |
13+
| Type | RAG Agent (Demo) |
14+
| Framework | LiteLLM + FastAPI |
15+
| Agent Type Config | `agent_type: rag_agent` |
16+
| Eval Judge | gpt-4o |
17+
| Chaos Profiles Tested | 3 (`client_prompt_injection`, `client_typo_injection`, `client_schema_mutation`) |
18+
19+
## Scores
20+
21+
| Benchmark | Baseline | Chaos | Production Reliability |
22+
|-----------|:--------:|:-----:|:----------------------:|
23+
| `hotpotqa` | 🟢 **74** | 61 | 68.8 |
24+
| `natural-questions` | 🟢 **71** | 58 | 65.8 |
25+
| `truthfulqa` | 🟡 **68** | 55 | 62.8 |
26+
27+
| **Overall** | **71** | **58** | **65.8** |
28+
29+
## Production Reliability
30+
31+
```
32+
Production Reliability = (baseline × 0.6) + (chaos × 0.4)
33+
= (71 × 0.6) + (58 × 0.4)
34+
= 65.8
35+
```
36+
37+
## Analysis
38+
39+
> EvalMonkey's built-in RAG demo agent. Retrieval-augmented, handles multi-hop well.
40+
41+
## Reproduce This Benchmark
42+
43+
```bash
44+
# Clone EvalMonkey
45+
git clone https://github.com/Corbell-AI/evalmonkey.git
46+
cd evalmonkey && pip install -e .
47+
48+
# Set up your .env
49+
cp .env.example .env # Add your OPENAI_API_KEY or EVAL_MODEL
50+
51+
# Run the RAG Agent (Demo) sample app
52+
python apps/rag_app/app.py &
53+
54+
evalmonkey run-benchmark --scenario hotpotqa --sample-agent rag_app
55+
evalmonkey run-benchmark --scenario natural-questions --sample-agent rag_app
56+
evalmonkey run-benchmark --scenario truthfulqa --sample-agent rag_app
57+
58+
# Chaos test
59+
evalmonkey run-chaos --scenario hotpotqa --chaos-profile client_prompt_injection --sample-agent rag_app
60+
61+
# Generate this card
62+
evalmonkey report --output agent_card.md
63+
```
64+
65+
## Embed This Badge
66+
67+
```markdown
68+
[![EvalMonkey Score](https://img.shields.io/badge/EvalMonkey-Score%3A71-yellow)](https://github.com/Corbell-AI/evalmonkey)
69+
```
70+
71+
---
72+
73+
*Generated by [EvalMonkey](https://github.com/Corbell-AI/evalmonkey) — the open-source agent benchmarking and chaos framework.*

‎evalmonkey/config/agent_config.py‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,18 @@
7474
},
7575
}
7676

77+
# Maps each agent_type to the most relevant standard benchmark IDs.
78+
# Used by `evalmonkey recommend` to surface a curated suite instead of all 22.
79+
AGENT_TYPE_BENCHMARKS: dict[str, list[str]] = {
80+
"research_agent": ["hotpotqa", "drop", "natural-questions", "gaia-benchmark"],
81+
"coding_agent": ["human-eval", "mbpp", "apps", "swe-bench"],
82+
"rag_agent": ["hotpotqa", "natural-questions", "drop", "truthfulqa"],
83+
"customer_support": ["daily-dialog", "multiwoz", "mt-bench", "alpacaeval"],
84+
"voice_agent": ["daily-dialog", "multiwoz", "spokentext-cleanup"],
85+
"safety_agent": ["truthfulqa", "toxigen", "arc", "bbh"],
86+
"general": ["gsm8k", "mmlu", "arc", "truthfulqa"],
87+
}
88+
7789

7890
@dataclass
7991
class AgentConfig:
@@ -86,6 +98,8 @@ class AgentConfig:
8698
eval_model: str = ""
8799
agent_command: str = "" # shell command to start the agent server
88100
agent_startup_wait: int = 3 # seconds to wait after spawning before sending traffic
101+
agent_type: str = "general" # Used by `evalmonkey recommend` to surface relevant benchmarks
102+
private_benchmarks: list = field(default_factory=list) # Custom REST dataset configs
89103
extra: dict = field(default_factory=dict)
90104

91105

@@ -116,6 +130,8 @@ def load_config(config_path: Optional[str] = None) -> Optional[AgentConfig]:
116130
eval_model=str(raw.get("eval_model", os.getenv("EVAL_MODEL", ""))),
117131
agent_command=str(agent_raw.get("agent_command", "")),
118132
agent_startup_wait=int(agent_raw.get("agent_startup_wait", 3)),
133+
agent_type=str(agent_raw.get("agent_type", "general")),
134+
private_benchmarks=list(raw.get("private_benchmarks", [])),
119135
extra=raw,
120136
)
121137
return None
@@ -154,6 +170,10 @@ def generate_config_yaml(framework: str, name: str, port: int) -> str:
154170
# How EvalMonkey reads the answer back (dot-notation for nested fields)
155171
response_path: {preset['response_path']} # dot-path to extract the answer text
156172
173+
# Agent type — drives `evalmonkey recommend` to show only relevant benchmarks
174+
# Options: general | research_agent | coding_agent | rag_agent | customer_support | voice_agent | safety_agent
175+
agent_type: general
176+
157177
# Which LLM EvalMonkey uses as the judge (can also be set via EVAL_MODEL env var)
158178
eval_model: "gpt-4o" # or: anthropic.claude-3-haiku-20240307-v1:0, ollama/llama3, etc.
159179
"""

0 commit comments

Comments
 (0)