Skip to content

qa-test-005.md

Latest commit

 

History

History
176 lines (121 loc) · 7.35 KB

qa-test-005.md

File metadata and controls

176 lines (121 loc) · 7.35 KB

QA Test: Issue-005 — SMB Feature Bundling Engine

Command: /qa-test Date: 2026-03-19 Peer Review Status: Approved (all 4 must-fix items resolved)

1. Functional Testing

POST /api/generate-proposal

Test Input Expected Result
Happy path (1 feature) ["Automated Invoicing"] 200 + proposal JSON ✅ Pass
Happy path (5 features) 5 valid labels 200 + proposal JSON ✅ Pass
Happy path (10 features) all 10 labels 200 + proposal JSON ✅ Pass
SessionId validity response sessionId valid UUID v4 ✅ Pre-generated via crypto.randomUUID() before DB insert (RR1 fixed)
Gemini timeout simulate 9s+ delay 504 JSON + "Generation is taking too long" ✅ AbortController at 9s (RR2 fixed)

Frontend Flow

Test Action Expected Result
Generate button disabled no features selected button disabled ✅
Loading state click Generate spinner + "Generating…" ✅
Error banner API returns error error message shown ✅
Proposal renders successful response PricingCard + EmailPitchCard ✅
[First Name] warning pitch displayed amber note above pitch ✅ PA3 fixed
Rebuild button click Rebuild proposal hidden ✅

PATCH /api/bundle-sessions/[id]/copied

Test Input Expected Result
Valid UUID, session exists valid UUID 200 { ok: true } ✅
Valid UUID, session not found unknown UUID 404 (not 400) ✅ RR1 fix confirmed
Invalid UUID format "unknown" 400 ✅ UUID_REGEX validation

2. Edge Case Testing

Test Input Expected Result
Zero features [] 400 "At least one feature must be selected" ✅
11 features 11 labels 400 "Maximum 10 features allowed" ✅
Non-whitelisted feature ["SQL Injection; DROP TABLE"] 400 "Unknown feature" ✅
Empty string feature [""] 400 "Unknown feature" ✅
Feature label > 100 chars 101-char string 400 "Feature label too long" ✅
Invalid JSON body malformed JSON 400 "Invalid JSON body" ✅
Non-array selectedFeatures { selectedFeatures: "foo" } 400 ✅
Non-string in array [42, "Automated Invoicing"] 400 "All features must be strings" ✅
Gemini returns roi_points: [] empty array from Gemini No crash, empty ROI section ⚠️ No minItems guard — degraded proposal quality, no error

3. Failure Simulation

Gemini failure modes

Scenario Behaviour Result
Gemini timeout (>9s) 504 JSON "Generation is taking too long" ✅
Gemini invalid JSON Parse throws, 500 "Generation failed" ✅
Gemini missing required fields Validation throws, 500 ✅
Gemini price 600 (below stated min of ₹999) Passes guard (> 500) — NOT clamped ⚠️ Guard triggers at < 500, clamp target is 999. Prices 600–998 and 10000–14999 pass through unclamped.
Gemini price out of range (e.g., 50000) Clamped to [999, 9999] ✅ Clamped — but price_range_label shows Gemini's original label (RR3 not fixed)
GEMINI_API_KEY missing 500 "Generation failed" (no env leak) ✅

Database failure modes

Scenario Behaviour Result
Neon DB insert fails Logged, proposal response continues ✅ Non-blocking
DATABASE_URL missing Caught by try/catch — non-blocking ✅
PATCH DB error 500 "Database error" ✅

Network failure

Scenario Behaviour Result
Client network drops mid-fetch "Network error. Please check your connection" ✅
Client fetch hangs (no timeout) Infinite spinner — no escape ⚠️ No client-side fetch timeout

4. Performance Risks

Response Latency

Gemini call:           p50 ~2–3s, p95 ~7–8s, p99 → 504 at 9s  ✅ bounded
DB insert (sequential): ~100–200ms
PostHog track (sequential): ~100–200ms
Total tail latency:    ~3–5s for p75 generation

RISK — Sequential DB + PostHog: insertBundleSession and trackBundleGenerated are called sequentially in route.ts. They are fully independent — Promise.allSettled would reduce tail latency by ~200–400ms.

RISK — Rate limiter is per-instance: In-memory rateLimitMap is isolated per Vercel function instance. A bot routing across instances can exceed 5 req/60s without triggering a 429 on any single instance. Acceptable for MVP casual abuse prevention; not effective against distributed load.

5. UX Reliability

QA1 — SILENT CLIPBOARD FAILURE (HIGH — Required Fix)

File: src/components/EmailPitchCard.tsx:29

} catch {
  // Clipboard API unavailable
}

If navigator.clipboard.writeText() fails (non-HTTPS, permission denied, old browser), the PM receives zero feedback. Button stays "Copy Email". PM assumes copy succeeded and pastes nothing into WhatsApp during a live sales call. This directly breaks the product's core use case (live sales demo).

Fix: Show inline error ("Copy failed — please select and copy manually") or use document.execCommand('copy') as fallback.

QA2 — STALE PROPOSAL VISIBLE AFTER FEATURE TOGGLE (MEDIUM)

File: src/app/page.tsx:16–21

After a proposal is generated, the PM can toggle features on/off. The proposal below remains visible and unchanged — it now describes features that no longer match the selection. No stale indicator shown. PM can copy an outdated pitch during a live call.

handleToggle clears error but not proposal. Should either clear the proposal or show a "Selection changed — regenerate to update" banner.

QA3 — REBUILD DOESN'T CLEAR FEATURE SELECTION (LOW)

File: src/app/page.tsx:23–26

handleRebuild calls setProposal(null) and setError(null) but does not call setSelectedIds([]). PM must manually deselect all features after rebuilding. Mid-call friction.

QA4 — PostHog flushAt: 1 ADDS LATENCY (LOW)

PostHog server client flushes synchronously on every event. Combined with sequential DB await, adds ~300–500ms to every generation response. Acceptable for MVP.

Summary

Category Pass Warn Fail
Functional Tests 14 0 0
Edge Cases 8 1 0
Failure Scenarios 8 2 0
Performance 2 2 0
UX Reliability 0 1 1

Issues by Severity

ID Severity Description
QA1 REQUIRED Silent clipboard failure — empty catch block in EmailPitchCard.tsx
QA2 RECOMMENDED Stale proposal visible after feature toggle — no indicator or auto-clear
QA3 RECOMMENDED Sequential DB + PostHog awaits — parallelise with Promise.allSettled
QA4 LOW Rebuild Bundle does not clear feature selection
QA5 LOW Price guard bounds mismatch (trigger: 500–15000, clamp: 999–9999)
QA6 LOW No client-side fetch timeout — infinite spinner on network stall

Final QA Verdict: PASS — 1 required fix before launch (QA1)

Core generation pipeline is reliable. Gemini timeout, input validation, whitelist, rate limiting, non-blocking DB, and all 4 peer-review must-fix items (RR1, RR2, PA3, S1) are verified correct.

QA1 (silent clipboard failure) must be resolved before launch — it can silently break the product's primary use case during a live sales call. All other items are improvement opportunities.