Skip to content

Commit 6a3e4cf

Browse files
rwilliamspbg-opsCopilotgithub-advanced-security[bot]
authored
Complete AI interaction review workflow (#99)
* chore: verify CI lint workflow and apply black/lint updates * Overhaul autonomy docs and harden CPU chaos validation - Add comprehensive autonomy core package (contracts, map/twin state, planner, predictor, readiness) with tests and integration points. - Extend backend with autonomy insight endpoints and align OpenAPI coverage. - Expand HUD and C2 surfaces with autonomy KPI, safety, recommendation, and timeline overlays. - Normalize drone telemetry ingest confidence/health metadata and map query contract support. - Switch node-agent to CPU-only PyTorch wheels to reduce container footprint in CI/dev runs. - Harden chaos suite fallback by passing admin auth headers for trigger_fl. - Fix monitoring package integration script/test wiring and refresh repository docs with a full autonomy operations guide plus updated README/C2/monitoring/changelog. * Add concrete AI interaction implementation plan - Turn the AI usability recommendations into a phased, ticketable plan. - Cover command bar, structured recommendations, approval flow, model routing, metadata, mission context, and search. - Define rollout criteria, validation strategy, and first-branch execution order for the new feature branch. * Add shared AI interaction summary and operator UX - add the /ai/interaction/summary backend payload with quick actions and recommendations - wire the HUD and C2 surfaces to the shared interaction summary - fix the app-shell fetch path, add focused C2 coverage, and refresh docs - update the AI interaction plan to record completed work and follow-up items * Complete AI interaction review workflow - add interaction history and decision logging endpoints on the backend - wire the HUD review drawer with approve, edit, reject, and undo flows - pass review metadata through the app shell and action callbacks - update C2, docs, and API specs to reflect the completed interaction plan * Enhance HUD digital twin lens Add digital twin lens panels for map freshness, route candidates, replay context, and envelope assumptions to strengthen operator twin perspective. * Potential fix for pull request finding 'CodeQL / Unreachable code' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Signed-off-by: Ryan <221235059+rwilliamspbg-ops@users.noreply.github.com> --------- Signed-off-by: Ryan <221235059+rwilliamspbg-ops@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
1 parent a25b612 commit 6a3e4cf

8 files changed

Lines changed: 958 additions & 76 deletions

File tree

‎AI_INTERACTION_CONCRETE_PLAN.md‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,13 +10,15 @@ Implemented so far:
1010

1111
1. Shared AI interaction summary endpoint for HUD and C2.
1212
2. Natural-language command bar and structured recommendation cards in the main HUD.
13-
3. Quick-action routing and summary previews in C2.
13+
3. Review drawer with approve/edit/reject/undo loops and interaction history.
14+
4. Quick-action routing and summary previews in C2.
15+
5. Structured interaction history and decision logging in the backend.
1416

1517
Remaining follow-up work:
1618

17-
1. Add explicit approve/edit/reject/undo loops for AI-suggested actions.
18-
2. Expand model-routing heuristics beyond the current summary/planner split.
19-
3. Persist user interaction history so operators can review prior AI decisions.
19+
1. Expand model-routing heuristics further for search and replay-heavy operator workflows.
20+
2. Add richer decision search and replay views on top of the persisted history log.
21+
3. Persist operator preferences for review drawer defaults and mode switches.
2022

2123
## Product Principles
2224

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ Traditional federated learning can look simple in a lab and fail badly in the fi
2828
- A small local node set exercises aggregation, policy checks, and proof verification.
2929
- The runtime exposes the same health path operators use in production-style demos.
3030
- The primary HUD and C2 view share a structured AI interaction summary so common actions stay explainable and easy to trigger.
31+
- The HUD now includes a review drawer and backend decision history so AI-suggested actions can be approved, edited, rejected, or undone with an audit trail.
3132

3233
## Try It Now
3334

‎docs/api/openapi.yaml‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -593,6 +593,48 @@ paths:
593593
type: object
594594
additionalProperties: true
595595

596+
/ai/interaction/history:
597+
get:
598+
tags: [Operations]
599+
summary: Get recent AI interaction review history
600+
operationId: getAiInteractionHistory
601+
parameters:
602+
- in: query
603+
name: limit
604+
schema:
605+
type: integer
606+
default: 50
607+
description: Maximum number of review records to return
608+
responses:
609+
'200':
610+
description: Recent AI interaction decisions
611+
content:
612+
application/json:
613+
schema:
614+
type: object
615+
additionalProperties: true
616+
617+
/ai/interaction/decision:
618+
post:
619+
tags: [Operations]
620+
summary: Record an AI interaction review decision
621+
operationId: recordAiInteractionDecision
622+
requestBody:
623+
required: true
624+
content:
625+
application/json:
626+
schema:
627+
type: object
628+
additionalProperties: true
629+
responses:
630+
'200':
631+
description: Review decision recorded
632+
content:
633+
application/json:
634+
schema:
635+
type: object
636+
additionalProperties: true
637+
596638
/model_registry:
597639
get:
598640
tags: [Operations]

‎frontend/src/App.jsx‎

Lines changed: 57 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,7 @@ function App() {
7070
const [health, setHealth] = useState(null);
7171
const [metricsSummary, setMetricsSummary] = useState(null);
7272
const [interactionSummary, setInteractionSummary] = useState(null);
73+
const [interactionHistory, setInteractionHistory] = useState([]);
7374
const [trustStatus, setTrustStatus] = useState(null);
7475
const [policyHistory, setPolicyHistory] = useState([]);
7576
const [founders, setFounders] = useState([]);
@@ -264,11 +265,12 @@ function App() {
264265
const fetchData = async () => {
265266
setLoading(true);
266267
try {
267-
const [hudFetch, healthFetch, metricsFetch, interactionFetch, foundersFetch, trainingFetch, opsHealthFetch, opsTrendsFetch] = await Promise.allSettled([
268+
const [hudFetch, healthFetch, metricsFetch, interactionFetch, interactionHistoryFetch, foundersFetch, trainingFetch, opsHealthFetch, opsTrendsFetch] = await Promise.allSettled([
268269
fetch(`${API_BASE}/hud_data`),
269270
fetch(`${API_BASE}/health`),
270271
fetch(`${API_BASE}/metrics_summary`),
271272
fetch(`${API_BASE}/ai/interaction/summary`),
273+
fetch(`${API_BASE}/ai/interaction/history?limit=20`),
272274
fetch(`${API_BASE}/founders`),
273275
fetch(`${API_BASE}/training/status`),
274276
fetch(`${API_BASE}/ops/health`),
@@ -287,11 +289,12 @@ function App() {
287289
}
288290
};
289291

290-
const [hud, healthData, metrics, interactionData, foundersData, trainingData, opsHealthData, opsTrendsData] = await Promise.all([
292+
const [hud, healthData, metrics, interactionData, interactionHistoryData, foundersData, trainingData, opsHealthData, opsTrendsData] = await Promise.all([
291293
safeJson(toResponse(hudFetch), hudData || {}),
292294
safeJson(toResponse(healthFetch), health || {}),
293295
safeJson(toResponse(metricsFetch), metricsSummary || {}),
294296
safeJson(toResponse(interactionFetch), interactionSummary || {}),
297+
safeJson(toResponse(interactionHistoryFetch), { decisions: interactionHistory || [] }),
295298
safeJson(toResponse(foundersFetch), founders || []),
296299
safeJson(toResponse(trainingFetch), { status: 'idle', active: false, round: 0 }),
297300
safeJson(toResponse(opsHealthFetch), opsHealth || null),
@@ -363,6 +366,7 @@ function App() {
363366
setHealth(healthData);
364367
setMetricsSummary(metrics);
365368
setInteractionSummary(interactionData);
369+
setInteractionHistory(Array.isArray(interactionHistoryData?.decisions) ? interactionHistoryData.decisions : []);
366370
setFounders(foundersData);
367371
setTrainingStatus(nextTraining);
368372
setOpsHealth(nextOpsHealth);
@@ -383,9 +387,18 @@ function App() {
383387
}
384388
};
385389

386-
const triggerFLRound = async () => {
390+
const triggerFLRound = async (reviewContext = {}) => {
387391
try {
388-
const response = await fetch(`${API_BASE}/trigger_fl`, { method: 'POST' });
392+
const response = await fetch(`${API_BASE}/trigger_fl`, {
393+
method: 'POST',
394+
headers: { 'Content-Type': 'application/json' },
395+
body: JSON.stringify({
396+
review_decision: reviewContext.decision || '',
397+
review_reason: reviewContext.reason || '',
398+
source_action_id: reviewContext.source_action_id || '',
399+
review_id: reviewContext.review_id || '',
400+
})
401+
});
389402
if (response.ok) {
390403
fetchData();
391404
}
@@ -395,7 +408,7 @@ function App() {
395408
}
396409
};
397410

398-
const startTraining = async (rounds = 0) => {
411+
const startTraining = async (rounds = 0, reviewContext = {}) => {
399412
try {
400413
const targetRounds = Number(rounds);
401414
const payload = Number.isFinite(targetRounds) && targetRounds > 0
@@ -404,7 +417,13 @@ function App() {
404417
const response = await fetch(`${API_BASE}/training/start`, {
405418
method: 'POST',
406419
headers: { 'Content-Type': 'application/json' },
407-
body: JSON.stringify(payload)
420+
body: JSON.stringify({
421+
...payload,
422+
review_decision: reviewContext.decision || '',
423+
review_reason: reviewContext.reason || '',
424+
source_action_id: reviewContext.source_action_id || '',
425+
review_id: reviewContext.review_id || '',
426+
})
408427
});
409428
if (!response.ok) {
410429
throw new Error(`Training start failed with ${response.status}`);
@@ -416,9 +435,18 @@ function App() {
416435
}
417436
};
418437

419-
const stopTraining = async () => {
438+
const stopTraining = async (reviewContext = {}) => {
420439
try {
421-
const response = await fetch(`${API_BASE}/training/stop`, { method: 'POST' });
440+
const response = await fetch(`${API_BASE}/training/stop`, {
441+
method: 'POST',
442+
headers: { 'Content-Type': 'application/json' },
443+
body: JSON.stringify({
444+
review_decision: reviewContext.decision || '',
445+
review_reason: reviewContext.reason || '',
446+
source_action_id: reviewContext.source_action_id || '',
447+
review_id: reviewContext.review_id || '',
448+
})
449+
});
422450
if (!response.ok) {
423451
throw new Error(`Training stop failed with ${response.status}`);
424452
}
@@ -429,9 +457,18 @@ function App() {
429457
}
430458
};
431459

432-
const createEnclave = async () => {
460+
const createEnclave = async (reviewContext = {}) => {
433461
try {
434-
const response = await fetch(`${API_BASE}/create_enclave`, { method: 'POST' });
462+
const response = await fetch(`${API_BASE}/create_enclave`, {
463+
method: 'POST',
464+
headers: { 'Content-Type': 'application/json' },
465+
body: JSON.stringify({
466+
review_decision: reviewContext.decision || '',
467+
review_reason: reviewContext.reason || '',
468+
source_action_id: reviewContext.source_action_id || '',
469+
review_id: reviewContext.review_id || '',
470+
})
471+
});
435472
const payload = await response.json().catch(() => ({}));
436473
setEnclaveActionMessage(payload?.message || `Enclave status: ${payload?.enclave_status || 'unknown'}`);
437474
fetchData();
@@ -461,7 +498,7 @@ function App() {
461498
}
462499
};
463500

464-
const submitVoiceQuery = async (overrideQuery) => {
501+
const submitVoiceQuery = async (overrideQuery, reviewContext = {}) => {
465502
const query = String(overrideQuery ?? voiceQuery ?? '').trim();
466503
if (!query) {
467504
return;
@@ -471,7 +508,13 @@ function App() {
471508
const response = await fetch(`${API_BASE}/chat`, {
472509
method: 'POST',
473510
headers: { 'Content-Type': 'application/json' },
474-
body: JSON.stringify({ query })
511+
body: JSON.stringify({
512+
query,
513+
review_decision: reviewContext.decision || '',
514+
review_reason: reviewContext.reason || '',
515+
source_action_id: reviewContext.source_action_id || '',
516+
review_id: reviewContext.review_id || '',
517+
})
475518
});
476519
if (response.ok) {
477520
const result = await response.json();
@@ -599,10 +642,12 @@ function App() {
599642
</header>
600643

601644
<HUD
645+
apiBase={API_BASE}
602646
hudData={hudData}
603647
health={health}
604648
metricsSummary={metricsSummary}
605649
interactionSummary={interactionSummary}
650+
interactionHistory={interactionHistory}
606651
trustStatus={trustStatus}
607652
policyHistory={policyHistory}
608653
founders={founders}

‎frontend/src/C2SwarmHUD.jsx‎

Lines changed: 25 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -146,13 +146,15 @@ const normalizeInteractionActions = (interactionSummary) => {
146146
kind: 'assistant_query',
147147
model_route: 'summary',
148148
requires_confirmation: false,
149+
undo_command: null,
149150
},
150151
{
151152
id: 'c2-run-epoch',
152153
label: 'Run Global FL Epoch',
153154
command: 'trigger_fl',
154155
kind: 'control_action',
155156
requires_confirmation: true,
157+
undo_command: null,
156158
},
157159
{
158160
id: 'c2-start-training',
@@ -161,6 +163,7 @@ const normalizeInteractionActions = (interactionSummary) => {
161163
parameters: { rounds: 10 },
162164
kind: 'control_action',
163165
requires_confirmation: true,
166+
undo_command: 'stop_training',
164167
},
165168
];
166169
};
@@ -175,7 +178,17 @@ const normalizeInteractionRecommendations = (interactionSummary, fallbackCount =
175178
: [];
176179
};
177180

178-
function C2SwarmHUD({ apiBase, interactionSummary }) {
181+
const normalizeInteractionHistory = (interactionHistory) => {
182+
if (Array.isArray(interactionHistory?.decisions)) {
183+
return interactionHistory.decisions;
184+
}
185+
if (Array.isArray(interactionHistory)) {
186+
return interactionHistory;
187+
}
188+
return [];
189+
};
190+
191+
function C2SwarmHUD({ apiBase, interactionSummary, interactionHistory }) {
179192
const [status, setStatus] = useState(null);
180193
const [mapState, setMapState] = useState(null);
181194
const [commandLog, setCommandLog] = useState([]);
@@ -207,6 +220,11 @@ function C2SwarmHUD({ apiBase, interactionSummary }) {
207220
[interactionSummary, commandLog.length]
208221
);
209222

223+
const interactionHistoryEntries = useMemo(
224+
() => normalizeInteractionHistory(interactionHistory),
225+
[interactionHistory]
226+
);
227+
210228
const fetchSnapshot = useCallback(async () => {
211229
try {
212230
const [statusResp, mapResp, commandsResp] = await Promise.all([
@@ -678,6 +696,12 @@ function C2SwarmHUD({ apiBase, interactionSummary }) {
678696
<div className="c2-submit-message">
679697
{(interactionSummary?.recommendations || interactionRecommendations).map((item) => `${item.label || item.action}: ${item.reason}`).join(' | ')}
680698
</div>
699+
<div className="c2-submit-message">
700+
{interactionHistoryEntries.slice(0, 5).map((entry) => `${entry.decision || 'recorded'} ${entry.action_label || entry.action_id || 'interaction'}${entry.reason ? `: ${entry.reason}` : ''}`).join(' | ')}
701+
</div>
702+
{!interactionHistoryEntries.length ? (
703+
<div className="c2-empty">No AI interaction decisions recorded yet.</div>
704+
) : null}
681705
<div className="c2-submit-message">
682706
{operatorAssistCards.map((card) => `${card.title}: ${card.detail}`).join(' | ')}
683707
</div>

0 commit comments

Comments
 (0)