Merge pull request #123 from WecoAI/dev

Merge Dev - Add langfuse integration and fix min/max score reporting (0.3.21)

Author: aliroberts

examples/langfuse-zeph-hr-qa/README.md

@@ -0,0 +1,95 @@
+# ZephHR QA — LangFuse + Weco Example
+
+Optimize a QA agent that answers HR policy questions over fictional ZephHR documentation.
+
+This example demonstrates using Weco with LangFuse as the evaluation backend. It uses LangFuse datasets, local code evaluators, and managed LLM-as-a-Judge evaluators configured in the LangFuse UI.
+
+## Prerequisites
+
+- Python 3.10+
+- `uv pip install 'weco[langfuse]' openai langfuse`
+- Environment variables:
+  ```bash
+  export OPENAI_API_KEY="..."
+  export LANGFUSE_SECRET_KEY="sk-lf-..."
+  export LANGFUSE_PUBLIC_KEY="pk-lf-..."
+  export LANGFUSE_BASE_URL="https://cloud.langfuse.com"  # or https://us.cloud.langfuse.com
+  ```
+
+## LangFuse UI Setup
+
+Before running optimization, configure two **managed evaluators** (LLM-as-a-Judge) in your LangFuse project. These run server-side and score each agent response automatically.
+
+1. Go to your project in [LangFuse](https://cloud.langfuse.com/) → **Evaluation** → **Evaluators**
+2. Click **+ New Evaluator** and create two evaluators:
+
+### Correctness evaluator
+
+- **Name**: `Correctness`
+- **Score**: 0 or 1 (binary factual accuracy)
+- **Variable mappings**:
+  - `{{input}}` → `$.input.question` (the user's question)
+  - `{{output}}` → `$.output.answer` (the agent's answer)
+  - `{{expected_output}}` → `$.expected_output.expected_answer` (the ground truth)
+
+### Helpfulness evaluator
+
+- **Name**: `Helpfulness`
+- **Score**: 0–1 continuous scale
+- **Variable mappings**:
+  - `{{input}}` → `$.input.question`
+  - `{{output}}` → `$.output.answer`
+
+> **Important:** Use the **live preview** when configuring each evaluator to verify the variable mappings are picking up the correct data from your traces. The evaluator names are case-sensitive — `Correctness` and `Helpfulness` must match exactly what you pass to `--langfuse-managed-evaluators`.
+
+The custom metric function `evaluators:qa_score` combines these scores locally: `Correctness * Helpfulness`.
+
+## Setup
+
+Create the LangFuse datasets:
+
+```bash
+cd examples/langfuse-zeph-hr-qa
+python setup_dataset.py
+```
+
+This creates two datasets: `zephhr-qa-opt` (optimization) and `zephhr-qa-holdout` (validation).
+
+## Optimize
+
+```bash
+weco run --source agent.py \
+  --eval-backend langfuse \
+  --langfuse-dataset zephhr-qa-opt \
+  --langfuse-target agent:answer_hr_question \
+  --langfuse-evaluators evaluators:json_schema_validity evaluators:conciseness \
+  --langfuse-managed-evaluators Correctness Helpfulness \
+  --langfuse-metric-function evaluators:qa_score \
+  --additional-instructions optimizer_exemplars.md \
+  --metric qa_score --goal maximize --steps 30
+```
+
+## Holdout Validation
+
+```bash
+weco run --source agent.py \
+  --eval-backend langfuse \
+  --langfuse-dataset zephhr-qa-holdout \
+  --langfuse-target agent:answer_hr_question \
+  --langfuse-evaluators evaluators:json_schema_validity evaluators:conciseness \
+  --langfuse-managed-evaluators Correctness Helpfulness \
+  --langfuse-metric-function evaluators:qa_score \
+  --metric qa_score --goal maximize --steps 1
+```
+
+## File Overview
+
+| File | Purpose |
+|------|---------|
+| `agent.py` | Baseline QA agent (gpt-4o-mini) — Weco optimizes the prompt |
+| `evaluators.py` | LangFuse-format evaluators + `qa_score` metric function |
+| `setup_dataset.py` | Idempotent LangFuse dataset creation from JSON |
+| `docs.md` | ZephHR product documentation (knowledge base) |
+| `optimizer_exemplars.md` | Few-shot Q&A examples passed via `--additional-instructions` |
+| `data/optimization_questions.json` | Optimization set (15 questions) |
+| `data/holdout_questions.json` | Held-out validation set (10 questions) |

examples/langfuse-zeph-hr-qa/agent.py

@@ -0,0 +1,62 @@
+"""Baseline QA agent for ZephHR documentation.
+
+Answers HR policy questions using only the docs.md knowledge base.
+Weco optimizes this file — specifically the SYSTEM_PROMPT and USER_TEMPLATE.
+"""
+
+import json
+from pathlib import Path
+
+from openai import OpenAI
+
+client = OpenAI()
+
+DOCS = Path(__file__).with_name("docs.md").read_text()
+
+SYSTEM_PROMPT = """You are a ZephHR support assistant. Answer the user's question
+using ONLY the provided documentation. Do not guess or invent policy details.
+
+If the documentation does not contain enough information to fully answer,
+say so clearly and state what IS covered.
+
+Return your answer as JSON with exactly these fields:
+- answer: your response to the question (string)
+- confidence: how confident you are the answer is fully supported by the docs (high/medium/low)
+- relevant_sections: list of section names from the docs you referenced"""
+
+USER_TEMPLATE = """Documentation:
+{docs}
+
+Question: {question}
+
+Return only JSON."""
+
+
+def answer_hr_question(inputs: dict) -> dict:
+    """Answer an HR policy question from the ZephHR docs."""
+    question = inputs.get("question", "")
+
+    response = client.chat.completions.create(
+        model="gpt-4o-mini",
+        messages=[
+            {"role": "system", "content": SYSTEM_PROMPT},
+            {"role": "user", "content": USER_TEMPLATE.format(docs=DOCS, question=question)},
+        ],
+        temperature=0.0,
+        response_format={"type": "json_object"},
+    )
+
+    try:
+        parsed = json.loads(response.choices[0].message.content)
+    except (TypeError, json.JSONDecodeError):
+        parsed = {}
+
+    confidence = parsed.get("confidence", "low")
+    if confidence not in ("high", "medium", "low"):
+        confidence = "low"
+
+    relevant_sections = parsed.get("relevant_sections", [])
+    if not isinstance(relevant_sections, list):
+        relevant_sections = []
+
+    return {"answer": parsed.get("answer", ""), "confidence": confidence, "relevant_sections": relevant_sections}

examples/langfuse-zeph-hr-qa/data/holdout_questions.json

@@ -0,0 +1,52 @@
+[
+  {
+    "id": "hold-001",
+    "question": "A new employee started 45 days ago and works full-time. Can they enroll in medical benefits yet?",
+    "expected_answer": "Not yet. Full-time employees become eligible for benefits on the first day of the month following 60 days of employment. At 45 days, the employee has not yet reached the 60-day threshold, so they must wait."
+  },
+  {
+    "id": "hold-002",
+    "question": "Can the support team's geofencing feature be used on the Professional plan?",
+    "expected_answer": "No. Geofencing for time and attendance is available only on the Enterprise plan."
+  },
+  {
+    "id": "hold-003",
+    "question": "An employee got married last month. Can they add their spouse to benefits outside of open enrollment?",
+    "expected_answer": "Yes. Marriage is a qualifying life event that allows benefits changes outside of open enrollment. The employee must submit a qualifying life event request within 30 days of the marriage."
+  },
+  {
+    "id": "hold-004",
+    "question": "What's the maximum number of PTO days a 7-year employee can accumulate before accrual stops?",
+    "expected_answer": "An employee with 6+ years of tenure earns 25 days per year. PTO accrual is capped at 1.5x the annual entitlement, which means the cap is 37.5 days. Once this balance is reached, accrual pauses until the balance drops below the cap."
+  },
+  {
+    "id": "hold-005",
+    "question": "Can a System Admin grant themselves the System Admin role?",
+    "expected_answer": "No. System admins cannot grant the System Admin role to themselves. Another System Admin must make this change. All permission changes are recorded in the audit log and retained for 7 years."
+  },
+  {
+    "id": "hold-006",
+    "question": "When is open enrollment and when do changes take effect?",
+    "expected_answer": "Open enrollment runs annually from November 1\u201315. Changes made during open enrollment take effect on January 1 of the following year."
+  },
+  {
+    "id": "hold-007",
+    "question": "We're on Enterprise. What integrations do we get that Professional doesn't have?",
+    "expected_answer": "Enterprise includes everything in Professional plus REST API access, custom webhooks, and HRIS data sync with Workday/SAP SuccessFactors. Enterprise also has a higher API rate limit of 500 requests per minute compared to Professional's 100 requests per minute."
+  },
+  {
+    "id": "hold-008",
+    "question": "If payroll is due on Saturday the 15th, when does the payment actually go out?",
+    "expected_answer": "If a payday falls on a weekend or public holiday, payment is issued on the preceding business day. So if the 15th is a Saturday, payment would be issued on Friday the 14th."
+  },
+  {
+    "id": "hold-009",
+    "question": "How many sick days does a part-time employee get and do they carry over?",
+    "expected_answer": "Part-time employees receive 5 sick days per year. Sick days reset on January 1 and do not roll over to the next year."
+  },
+  {
+    "id": "hold-010",
+    "question": "An employee has been on sick leave for 4 consecutive days. Is there anything extra they need to do?",
+    "expected_answer": "Yes. Sick leave beyond 3 consecutive days requires a doctor's note uploaded through the portal."
+  }
+]

examples/langfuse-zeph-hr-qa/data/optimization_questions.json

@@ -0,0 +1,77 @@
+[
+  {
+    "id": "opt-001",
+    "question": "An employee was rehired 10 months after leaving. Do they need to redo full onboarding?",
+    "expected_answer": "No. Employees rehired within 12 months of their termination date may use a streamlined re-hire flow that preserves their previous tax elections. Since 10 months is within the 12-month window, they qualify for the streamlined process."
+  },
+  {
+    "id": "opt-002",
+    "question": "Our company is on the Starter plan. Can we set up SSO?",
+    "expected_answer": "SSO/SAML is not available on the Starter plan. It is available on the Professional plan as an add-on at $2 per user per month, and it is included with the Enterprise plan."
+  },
+  {
+    "id": "opt-003",
+    "question": "I'm a manager. Can I see how much my direct report makes?",
+    "expected_answer": "No. Managers cannot view or edit compensation details for their reports. Only HR admins have access to compensation information."
+  },
+  {
+    "id": "opt-004",
+    "question": "We need to issue a bonus to an employee. What's the process?",
+    "expected_answer": "Bonuses are processed as off-cycle payments. An HR admin must request the off-cycle payment, and it requires VP-level approval. Once approved, the off-cycle run is processed within 3 business days."
+  },
+  {
+    "id": "opt-005",
+    "question": "When does a new full-time employee become eligible for medical benefits?",
+    "expected_answer": "Full-time employees (30+ hours/week) become eligible for benefits on the first day of the month following 60 days of employment."
+  },
+  {
+    "id": "opt-006",
+    "question": "What is the SLA for a broken SSO integration?",
+    "expected_answer": "SSO failures are classified as P2 \u2013 High priority. The response time SLA is 4 hours with a resolution target of 1 business day."
+  },
+  {
+    "id": "opt-007",
+    "question": "Can an HR admin change their own salary in the system?",
+    "expected_answer": "No. HR admins cannot modify their own compensation or benefits elections. Another HR admin must make the change. All permission changes are recorded in the audit log."
+  },
+  {
+    "id": "opt-008",
+    "question": "An hourly employee forgot to submit their timesheet by Monday 5 PM. When will they get paid?",
+    "expected_answer": "Timesheet submissions for hourly employees must be approved by the direct manager by 5:00 PM local time on Monday of the pay week. Late submissions are processed in the next pay cycle with no exceptions."
+  },
+  {
+    "id": "opt-009",
+    "question": "We're on the Professional plan. What's our API rate limit?",
+    "expected_answer": "On the Professional plan, the API rate limit is 100 requests per minute."
+  },
+  {
+    "id": "opt-010",
+    "question": "An employee who worked 32 hours/week just dropped to 28 hours/week for 2 months. Do they lose medical?",
+    "expected_answer": "Not yet. Employees who drop below 30 hours/week lose medical benefits only after 3 consecutive months below the threshold. After 2 months, they still retain medical benefits. If they remain below 30 hours for a third consecutive month, they will be reclassified as part-time and lose medical benefits at the end of the third month, but they will retain dental and vision."
+  },
+  {
+    "id": "opt-011",
+    "question": "Can a department block PTO for 6 weeks during a product launch?",
+    "expected_answer": "No. Departments may declare blackout periods of a maximum of 4 weeks per year. A 6-week blackout would exceed the maximum allowed duration. Blackout periods must also be announced at least 30 days in advance."
+  },
+  {
+    "id": "opt-012",
+    "question": "Does ZephHR handle payroll taxes for our employees in the UK?",
+    "expected_answer": "ZephHR automatically calculates payroll taxes only for US and Canadian employees. For employees in other jurisdictions like the UK, payroll tax calculations must be configured manually by an HR admin with the Payroll Configuration permission."
+  },
+  {
+    "id": "opt-013",
+    "question": "How long is COBRA coverage and who manages it?",
+    "expected_answer": "COBRA coverage extends for up to 18 months. COBRA administration is handled by the third-party vendor HealthBridge, not directly through ZephHR. COBRA continuation is offered within 14 days of the employee's termination date."
+  },
+  {
+    "id": "opt-014",
+    "question": "I want to request 5 days of PTO starting next week. What approvals do I need?",
+    "expected_answer": "Requests of 4 or more days require approval from both your direct manager and the department head. Additionally, planned absences must be submitted at least 5 business days in advance."
+  },
+  {
+    "id": "opt-015",
+    "question": "We want to downgrade from Professional to Starter mid-contract. Will we get a refund?",
+    "expected_answer": "Downgrading from Professional to Starter mid-contract forfeits access to Professional features immediately with no prorated refund."
+  }
+]