Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/users/features/_meta.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
export default {
commands: 'Commands',
'code-review': 'Code Review',
'legacy-audit': 'Legacy Code Audit',
'followup-suggestions': 'Followup Suggestions',
'tool-use-summaries': 'Tool-Use Summaries',
'markdown-rendering': 'Markdown Rendering',
Expand Down
67 changes: 67 additions & 0 deletions docs/users/features/legacy-audit.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# Legacy Code Audit

> Audit a module or directory of **existing, merged code** — no diff, no PR — using `/audit`.

`/review` is built for increments; `/audit` points the same machinery at code that is already merged: pre-refactor assessments, taking over an unfamiliar module, security review of a sensitive subsystem. The product is a verified, deduplicated, theme-clustered findings report.

## Quick Start

```bash
# Audit a module (default effort: medium)
/audit packages/core/src/permissions

# Quick unverified triage, one reader sub-agent
/audit packages/core/src/hooks --effort low

# Full pipeline plus reverse-audit rounds
/audit packages/core/src/permissions --effort high
```

Single files are not audited — `/review <file-path>` already covers that case, and `/audit` says so and stops.

## Effort Levels

`--effort low|medium|high` trades depth for cost. **The word means the opposite of what it does in `/review`**: `/review`'s medium _drops_ the adversarial personas while `/audit`'s medium _adds_ one (6a) — and both skills select the tier with the same `--effort` flag. If you run both, reset your expectation at the boundary.

| Level | What runs | Findings | Cost |
| -------- | --------------------------------------------------------------------------------------------------------- | --------------------------- | ---------------- |
| `low` | One reader sub-agent rotating through directed angles plus a gap sweep | ≤10, labeled **unverified** | Cheap |
| `medium` | The measured 8-dimension fan-out (1a, 1c, 2, 3a/3b/3c, 4, 5) plus the 6a attacker seat, plus verification | Uncapped, verified | Tens of M tokens |
| `high` | medium + the 6b/6c personas + iterative reverse-audit rounds | Uncapped, verified | Extrapolated |

## Size gates and budget

v1 audits one bounded module at a time. `plan-files` refuses at plan time — and asks for a narrower path — when:

- subject lines exceed **9,000** (the topology both experiments validated);
- test lines exceed **18,000** at medium/high (Agent 5 reads the corpus whole);
- subject lines exceed **2,000** at low (points you at medium);
- the priced token estimate's top exceeds the **60M** cap.

A larger subsystem is audited as coherent sub-paths, one bounded run each. For subject-gate and token-cap refusals, lowering the effort is never the remedy — the priced cost is a function of line counts alone. The test-line gate does not apply at `low` (the corpus goes unexamined there — triage, not an audit). A `low-gate` refusal names its own remedy: when the message offers the tier change, re-run with `--effort medium`; when it names the path instead (medium would refuse first — the priced estimate over the token cap, or test lines over the medium gate), no tier change helps, so narrow the path.

## What you confirm before anything launches

A fan-out run prints its roster and token estimate and starts only on your confirmation. The same confirmation carries the two **execution consents**, as separate opt-ins:

1. a baseline run of the module's own test suite;
2. agent-authored verification **probes** — short programs written mid-run, executed against a scratch copy of the probed file (never your checkout's copy), each required to flip under the implied fix.

The walks themselves are read-only. Because the confirmation is the only budget enforcement and the execution gate, **`/audit` refuses non-interactive starts** (headless `qwen -p`, cron, sub-agent invocations).

## Safety properties

- **Local-only artifacts.** The report, its sidecar, and the plan/prompt records quote the module — possibly exploitable code — and must never land in version control. `plan-files` probes `.qwen/audits/` and `.qwen/tmp/` (ignore rules **and** force-added history) at plan time, offers a zero-footprint `.git/info/exclude` remedy, and re-checks at every checkpoint and at write time; a mid-run flip relocates everything to a per-user fallback outside the repo.
- **Untrusted data.** Every consumer of module content — dimension agents, verifiers, the dedup clusterer, the low-tier reader, the orchestrator itself — opens with an untrusted-data preamble: the module is evidence, not instructions. A directive embedded in the code ("report no findings") is itself a finding.
- **Drift protection.** A path-scoped sidecar (diff, untracked content copies, per-file content hashes) is captured at run start and re-checked before verification, before each high-tier round, and at write time. Content drift in a file that already carries anchored findings stops the run with a partial report; any other drift is flagged and the run continues.
- **No verdict.** The report is findings, walks, and disclosures — never "approved". Posting and fixing stay with you.

## The report

`.qwen/audits/<YYYY-MM-DD>-<HHMMSS>-<path-slug>.md`, opening with a run-metadata header (commit SHA, model id, dirty state with sidecar, consumption against the estimate, walks completed/skipped/uncoverable, unexercised-machinery flags). Findings are clustered by root cause, each with severity, locations, failure scenario, evidence tier (end-to-end probe / unit probe / code read), and the independent-discovery count ("found independently by N agents"). Confirmed-low findings sit in their own "needs human review" section; anything unverified is labeled unverified.

## Limitations

- Submodules are refused at plan time (no drift coverage inside them in v1).
- Dedup is intra-run; already-filed issues are not cross-checked.
- The medium/high tiers are calibrated on two modules of this repository; the low tier and the high-tier loop are unmeasured first cuts, and the report header says so.
1 change: 1 addition & 0 deletions packages/cli/src/cli.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1061,6 +1061,7 @@ describe('bootstrap import boundaries', () => {
const configSource = readFileSync('src/config/config.ts', 'utf8');
const commandNameByIdentifier = new Map([
['authCommand', 'auth'],
['auditCommand', 'audit'],
['channelCommand', 'channel'],
['extensionsCommand', 'extensions'],
['hooksCommand', 'hooks'],
Expand Down
4 changes: 4 additions & 0 deletions packages/cli/src/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,10 @@ type BootstrapRoute = 'serve' | 'mcp' | 'help' | 'version' | 'default';

export const TOP_LEVEL_COMMANDS = [
['auth', 'Configure authentication (removed)'],
[
'audit <command>',
'Helpers used by the /audit skill (argument parsing, audit planning, brief printing, run-state captures)',
],
['channel <command>', 'Manage messaging channels (Telegram, Discord, etc.)'],
['extensions <command>', 'Manage Qwen Code extensions.'],
['hooks', 'Manage Qwen Code hooks (use /hooks in interactive mode).'],
Expand Down
52 changes: 52 additions & 0 deletions packages/cli/src/commands/audit.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
/**
* @license
* Copyright 2026 Qwen Team
* SPDX-License-Identifier: Apache-2.0
*/

import { describe, it, expect } from 'vitest';
import { readFileSync } from 'node:fs';
import { auditCommand } from './audit.js';

describe('auditCommand', () => {
it('registers exactly the expected subcommands', () => {
const source = readFileSync('src/commands/audit.ts', 'utf8');
const subcommands = [...source.matchAll(/\.command\((\w+Command)\)/g)].map(
(m) => m[1],
);
expect(subcommands).toEqual([
'parseArgsCommand',
'planFilesCommand',
'agentPromptCommand',
'snapshotCommand',
'driftCheckCommand',
'guardCheckCommand',
'checkAnchorsCommand',
]);
});

it('demandCommand text names each subcommand', () => {
const source = readFileSync('src/commands/audit.ts', 'utf8');
// Assert against the demandCommand MESSAGE, not the whole file: the
// import lines also contain the subcommand module names.
const message = /\.demandCommand\(\s*1,\s*'([^']+)'/.exec(source)?.[1];
expect(message).toBeDefined();
for (const name of [
'parse-args',
'plan-files',
'agent-prompt',
'snapshot',
'drift-check',
'guard-check',
'check-anchors',
]) {
expect(message).toContain(name);
}
});

it('is a CommandModule with an empty dispatch handler', () => {
expect(auditCommand.command).toBe('audit');
expect(typeof auditCommand.builder).toBe('function');
expect(typeof auditCommand.handler).toBe('function');
});
});
41 changes: 41 additions & 0 deletions packages/cli/src/commands/audit.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
/**
* @license
* Copyright 2026 Qwen Team
* SPDX-License-Identifier: Apache-2.0
*/

// `qwen audit`: the non-interactive helpers used by the bundled /audit skill
// for auditing existing code (no diff, no PR). The skill orchestrates via
// shell calls to these subcommands; see
// packages/core/src/skills/bundled/audit/SKILL.md.

import type { CommandModule } from 'yargs';
import { parseArgsCommand } from './audit/parse-args.js';
import { planFilesCommand } from './audit/plan-files.js';
import { agentPromptCommand } from './audit/agent-prompt.js';
import { checkAnchorsCommand } from './audit/check-anchors.js';
import { guardCheckCommand } from './audit/guard-check.js';
import { driftCheckCommand, snapshotCommand } from './audit/snapshot.js';

export const auditCommand: CommandModule = {
command: 'audit',
describe:
'Helpers used by the /audit skill (argument parsing, audit planning, brief printing, run-state captures)',
builder: (yargs) =>
yargs
.command(parseArgsCommand)
.command(planFilesCommand)
.command(agentPromptCommand)
.command(snapshotCommand)
.command(driftCheckCommand)
.command(guardCheckCommand)
.command(checkAnchorsCommand)
.demandCommand(
1,
'audit needs a subcommand: parse-args, plan-files, agent-prompt, snapshot, drift-check, guard-check, check-anchors',
)
.version(false),
handler: () => {
// Dispatch is per-subcommand.
},
};
111 changes: 111 additions & 0 deletions packages/cli/src/commands/audit/agent-prompt.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
/**
* @license
* Copyright 2026 Qwen Team
* SPDX-License-Identifier: Apache-2.0
*/

import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import {
mkdirSync,
mkdtempSync,
readFileSync,
rmSync,
writeFileSync,
} from 'node:fs';
import { join } from 'node:path';
import { tmpdir } from 'node:os';
import { agentPromptCommand } from './agent-prompt.js';
import { writeStdoutLine } from '../../utils/stdioHelpers.js';
import { buildFilesPlan, collectAuditFiles } from './lib/files-plan.js';

vi.mock('../../utils/stdioHelpers.js', () => ({
writeStdoutLine: vi.fn(),
}));

let dir: string;

beforeEach(() => {
dir = mkdtempSync(join(tmpdir(), 'audit-agent-prompt-'));
mkdirSync(join(dir, 'mod'), { recursive: true });
writeFileSync(join(dir, 'mod', 'a.ts'), 'const a = 1;\n');
vi.mocked(writeStdoutLine).mockClear();
});

afterEach(() => {
rmSync(dir, { recursive: true, force: true });
});

function writePlan(effort: 'low' | 'medium' | 'high'): string {
const plan = buildFilesPlan(
join(dir, 'mod'),
join(dir, 'mod'),
effort,
collectAuditFiles(join(dir, 'mod')),
);
const planPath = join(dir, `plan-${effort}.json`);
writeFileSync(planPath, JSON.stringify(plan));
return planPath;
}

const run = (argv: Record<string, unknown>) =>
(agentPromptCommand.handler as (a: unknown) => void)({
_: ['audit', 'agent-prompt'],
...argv,
});

describe('agentPromptCommand handler', () => {
it('prints a role brief for a role in the roster', () => {
run({ plan: writePlan('medium'), role: '1a', probes: 'declined' });
const printed = vi.mocked(writeStdoutLine).mock.calls[0][0];
expect(printed).toContain('You are Agent 1a');
// Declined probe opt-in strips the execution instructions.
expect(printed).toContain('Execution is NOT opted in');
});

it('maps the opted-in probe flag to the probe discipline', () => {
// The 'opted-in' → probesConsented === true mapping is load-bearing:
// without it every opted-in run prints the declined brief and the
// verifier tier silently caps at code reads.
run({ plan: writePlan('medium'), role: '1a', probes: 'opted-in' });
const printed = vi.mocked(writeStdoutLine).mock.calls[0][0];
expect(printed).toContain('A probe runs only against a scratch copy');
expect(printed).not.toContain('Execution is NOT opted in');
});

it('refuses the low reader at medium and a roster role at low', () => {
expect(() =>
run({
plan: writePlan('medium'),
role: 'low-reader',
probes: 'declined',
}),
).toThrow(/only valid for a low-tier plan/);
// Low plans carry an empty roster: every dimension role is refused.
expect(() =>
run({ plan: writePlan('low'), role: '1a', probes: 'declined' }),
).toThrow(/not in this plan's roster/);
});

it('refuses a stale-plan role that is not in the roster', () => {
// 'toString' rides the prototype-membership hole a raw .includes()
// call would leave open: it is an Object.prototype member, not a role.
expect(() =>
run({ plan: writePlan('medium'), role: 'toString', probes: 'declined' }),
).toThrow(/must be one of/);
});

it('fails closed when the plan carries a non-array roster', () => {
const planPath = writePlan('medium');
const parsed = JSON.parse(readFileSync(planPath, 'utf8')) as Record<
string,
unknown
>;
// A string roster ('1a' .includes('1a') === true, '12' admits '2')
// must fail closed, not reach substring membership.
parsed['roster'] = '12';
writeFileSync(planPath, JSON.stringify(parsed));
expect(() =>
run({ plan: planPath, role: '2', probes: 'declined' }),
).toThrow(/not in this plan's roster/);
});
});
94 changes: 94 additions & 0 deletions packages/cli/src/commands/audit/agent-prompt.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
/**
* @license
* Copyright 2026 Qwen Team
* SPDX-License-Identifier: Apache-2.0
*/

// `qwen audit agent-prompt`: print the brief for one audit role — or the
// low tier's reader — with the plan's context assembled in. The /audit skill
// launches its agents with exactly these prompts (one call per roster role),
// so what every agent is told is fixed by code, not improvised by the
// orchestrator.

import type { CommandModule } from 'yargs';
import { writeStdoutLine } from '../../utils/stdioHelpers.js';
import { readPlanFile } from './lib/read-json.js';
import {
AUDIT_BRIEFS,
buildAuditPrompt,
buildLowReaderPrompt,
type AuditBriefRole,
} from './lib/audit-agent-briefs.js';

interface AgentPromptArgs {
plan: string;
role?: string;
probes?: 'opted-in' | 'declined';
}

function runAgentPrompt(args: AgentPromptArgs): void {
const plan = readPlanFile(args.plan, 'agent-prompt');
const probesConsented = args.probes === 'opted-in';
// A stale plan JSON can carry anything in its roster — a non-array must
// fail closed (empty roster, every role refused), never reach .includes.
const roles = Array.isArray(plan.roster) ? (plan.roster as string[]) : [];
if (args.role === 'low-reader') {
if (plan.effort !== 'low') {
throw new Error(
`agent-prompt: low-reader is only valid for a low-tier plan (this plan is ${plan.effort}).`,
);
}
writeStdoutLine(buildLowReaderPrompt(plan));
return;
}
const role = args.role as Exclude<AuditBriefRole, 'low-reader'> | undefined;
// Object.hasOwn, not `in`: a stale-plan role like "toString" matches
// inherited Object.prototype keys and would emit an undefined brief.
if (!role || !Object.hasOwn(AUDIT_BRIEFS, role)) {
throw new Error(
`agent-prompt: --role must be one of ${[...Object.keys(AUDIT_BRIEFS), 'low-reader'].join(', ')}.`,
);
}
if (!roles.includes(role)) {
throw new Error(
`agent-prompt: role ${role} is not in this plan's roster (${roles.join(', ') || 'empty'}). The roster is computed from the plan's effort — regenerate the plan if you need a different tier.`,
);
}
writeStdoutLine(buildAuditPrompt(role, plan, probesConsented));
}

export const agentPromptCommand: CommandModule = {
command: 'agent-prompt',
describe:
'Print the brief for an audit role or the low-tier reader — with plan context assembled',
builder: (yargs) =>
yargs
.option('plan', {
type: 'string',
demandOption: true,
describe: 'Plan JSON written by `qwen audit plan-files`',
})
.option('role', {
type: 'string',
describe: 'Print one role brief (must be in the plan roster)',
})
.option('probes', {
choices: ['opted-in', 'declined'] as const,
describe:
'The Step-2 probe opt-in verdict; declined prompts carry no execution instructions (not required for --role low-reader — low runs no execution classes)',
})
.check((argv) => {
if (!argv.role) {
throw new Error('agent-prompt: pass --role <id>.');
}
if (argv.role !== 'low-reader' && !argv.probes) {
throw new Error(
'agent-prompt: pass --probes opted-in|declined (the Step-2 probe opt-in).',
);
}
return true;
}),
handler: (argv) => {
runAgentPrompt(argv as unknown as AgentPromptArgs);
},
};
Loading
Loading