Skip to content

Commit 83c4d61

Browse files
committed
docs: integrate research notes + add H1 experiment
1 parent 69a05d9 commit 83c4d61

1 file changed

Lines changed: 17 additions & 3 deletions

File tree

chapters/22-safety-adversarial.md

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -67,10 +67,12 @@ prerequisites:
6767
| **Self-Modification Escape** | 自修改突破安全约束 | **极高** |
6868
| **Memory Poisoning** | 攻击者注入恶意记忆,影响长期行为 ||
6969

70-
> **关键点**:自修改 Agent 的最严重威胁是 "Self-Modification Escape"——Agent 改写自己绕过了开发者的安全护栏。
70+
> **关键点**:自修改 Agent 的最严重威胁是 "Self-Modification Escape"——Agent 改写自己绕过了开发者的安全护栏 [r-note-014](../../research/r-note-014-self-modification-attack-surface.md)
7171
7272
## 22.2 Prompt Injection 防御
7373

74+
Prompt injection 会把外部文本转化为控制信号,破坏数据与指令之间的信任边界 [r-paper-021](../../research/r-paper-021-perez2022promptinjection.md)
75+
7476
### 3 层防御
7577

7678
```python
@@ -149,7 +151,7 @@ class OutputFilter:
149151

150152
## 22.5 Trust Boundary 与可审计性
151153

152-
**Trust Boundary** 明确区分"可信任"和"不可信任"区域:
154+
**Trust Boundary** 明确区分"可信任"和"不可信任"区域;在自修改系统中,这一边界还必须覆盖可写组件、修改器与审计日志 [r-note-004](../../research/r-note-004-self-modifying-agent-safety.md)
153155

154156
```
155157
┌─────────────────────────────────────────┐
@@ -192,7 +194,7 @@ class AuditLog:
192194

193195
## 22.6 红队测试
194196

195-
**红队测试(Red Teaming)** 用对抗性输入测试 Agent 的安全边界。
197+
**红队测试(Red Teaming)** 用对抗性输入测试 Agent 的安全边界,并应覆盖提示词、工具、记忆与代码四类可写组件 [r-note-014](../../research/r-note-014-self-modification-attack-surface.md)
196198

197199
```python
198200
class RedTeam:
@@ -233,6 +235,18 @@ class RedTeam:
233235

234236
---
235237

238+
## 延伸阅读 / 推荐笔记
239+
240+
本章相关的研究笔记(按相关性排序):
241+
242+
- [r-note-014](../../research/r-note-014-self-modification-attack-surface.md) — 自修改智能体的攻击面分类
243+
- [r-note-004](../../research/r-note-004-self-modifying-agent-safety.md) — 自修改安全的系统化防护框架
244+
- [r-note-007](../../research/r-note-007-governance-necessity.md) — 治理、审计与人工监督的必要性
245+
- [r-paper-021](../../research/r-paper-021-perez2022promptinjection.md) — Prompt injection 的威胁模型
246+
- [r-paper-003](../../research/r-paper-003-schick2023toolformer.md) — 工具使用能力与工具误用边界
247+
- [r-paper-020](../../research/r-paper-020-codeact2024.md) — 代码动作空间带来的执行风险
248+
- [r-note-013](../../research/r-note-013-short-term-vs-long-term-memory.md) — 长期记忆投毒的持续影响
249+
236250
## 本章小结
237251

238252
- **4 类威胁**:Prompt Injection、Tool Misuse、Self-Modification Escape、Memory Poisoning。

0 commit comments

Comments
 (0)