diff --git a/README.md b/README.md index 8f73b50c..58ad3e84 100644 --- a/README.md +++ b/README.md @@ -27,6 +27,74 @@ https://discord.com/invite/Wfr2ejBmY > **Support continued Engraphis development with Pro.** [Start a 3-day Pro trial](https://api.engraphis.com/account?plan=pro&interval=monthly&utm_source=engraphis&utm_medium=docs&utm_campaign=pro_conversion&utm_content=readme_intro&trial=pro#billing) > or [subscribe to Pro](https://api.engraphis.com/account?plan=pro&interval=monthly&utm_source=engraphis&utm_medium=docs&utm_campaign=pro_conversion&utm_content=readme_intro#billing). +--- + +## Measured token and context savings + +

+ Dark chart showing Engraphis using 98.21 percent less long-history context, 73.0 percent less retrieved content per question, 73.9 percent fewer tokens in the smallest useful memory, a 55.38 percent smaller memory response, and 47.8 percent less repeated-memory context after consolidation +
+ Less repeated history means more room for the task, tools, and useful evidence. +

+ +
+See benchmark details and reproduce the results + +### Controlled before-and-after example + +| Retrieval mode | Mean returned memory content | Recall@5 | +|---|---:|---:| +| Whole documents | 808.8 tokens | 1.000 | +| Engraphis structure-aware chunks | 218.4 tokens | 1.000 | + +The chunked mode returns the relevant passage instead of the whole document: **590.4 fewer tokens +per question**. Under the same model-context budget, that leaves roughly **590 tokens** for task +instructions or other relevant evidence. + +### Measurement details and reproducibility + +The table below records every current token/context efficiency measurement and its counting +boundary. + +| What is counted | Comparison | Measured reduction | Quality held constant | +|---|---|---|---| +| Cumulative reader context across a 1,986-question LoCoMo diagnostic | Full-history replay: **49,915,394** tokens → Engraphis: **891,857** tokens | **49,023,537 fewer context tokens** (**98.2133% lower**) | Focused retrieval used far less context; uncapped full history retained higher retrieval recall | +| Retrieved top-5 memory content, averaged per question | Whole documents: **808.8** tokens → structure-aware chunks: **218.4** tokens | **590.4 fewer tokens per question** (**73.0% lower**, about **3.7× smaller**) | Recall@5 **1.000** in both modes across 6 documents and 18 questions | +| Smallest returned memory that contains the reference evidence | Whole documents: **162.2** tokens → chunks: **42.4** tokens | **119.8 fewer tokens to evidence** (**73.9% lower**, about **3.8× smaller**) | The same 18 questions had a returned evidence-holding memory in both modes | +| Serialized MCP recall response across 260 timed CodeMem recalls | Full result: **17,172** `engraphis.regex.v1` tokens → compact result: **7,663** tokens | **9,509 response tokens avoided** (**55.38% lower**) | Recall@5, hit@5, and answer-token recall all **1.000** | +| Repeated-memory consolidation fixture | 12 related episodic memories: **230** tokens → one digest: **120** tokens | **110 tokens removed from the active digest** (**47.8% lower**) | Original memories remain available for provenance and audit | +| Packed prompt-context usage in the same CodeMem performance fixture | Hard budget: **1,500** tokens; observed mean: **87.73**; observed maximum: **106** | A hard cap prevents a recall from exceeding its configured context budget | This is usage accounting, not a before/after savings comparison | + +The compact MCP response avoids duplicating full memory bodies when the packed context and source +list are enough. That can reduce what an agent must inspect or pass onward, but the fixtures do +**not** measure model-provider charges, end-to-end task time, or customer cost savings. + +The measures are deliberately separate and **must not be added together**: chunking counts the +content of retrieved memory records before `ContextPacker`, whereas compact recall counts the +serialized MCP response returned to a client. “Tokens to evidence” is the size of the smallest +retrieved memory record holding the reference evidence; it is not latency or end-to-end answer +accuracy. Chunking creates more focused stored records (24 chunks rather than 6 whole-document +memories in this fixture), so this is a context-efficiency result, not a storage-reduction claim. + +Reproduce the quality and token/context measurements without a network connection or API key: + +```bash +python -m eval.harness --dataset eval/datasets/codemem.jsonl --k 5 +python -m eval.grounded +python -m eval.chunking_eval +python -m eval.performance --dataset eval/datasets/codemem.jsonl --k 5 --iterations 10 --json +``` + +These are small deterministic correctness and efficiency fixtures, not official LoCoMo / +LongMemEval QA scores or a third-party leaderboard result. Compact-response counts use the exact +`engraphis.regex.v1` counter; the chunking evaluation uses its documented deterministic +normalized-character estimator. Chunking measures retrieved memory content, while compact recall +measures serialized MCP response size. See [`BENCHMARKS.md`](BENCHMARKS.md) for definitions, +limitations, canonical external-evaluation requirements, and the no-unsupported-claims policy. + +
+ +--- ## What Engraphis gives an agent @@ -130,73 +198,6 @@ chunking. The activity view records outcomes, never keys, prompts, or raw provid --- -## Measured token and context savings - -

- Dark chart showing Engraphis using 98.21 percent less long-history context, 73.0 percent less retrieved content per question, 73.9 percent fewer tokens in the smallest useful memory, a 55.38 percent smaller memory response, and 47.8 percent less repeated-memory context after consolidation -
- Less repeated history means more room for the task, tools, and useful evidence. -

- -
-See benchmark details and reproduce the results - -### Controlled before-and-after example - -| Retrieval mode | Mean returned memory content | Recall@5 | -|---|---:|---:| -| Whole documents | 808.8 tokens | 1.000 | -| Engraphis structure-aware chunks | 218.4 tokens | 1.000 | - -The chunked mode returns the relevant passage instead of the whole document: **590.4 fewer tokens -per question**. Under the same model-context budget, that leaves roughly **590 tokens** for task -instructions or other relevant evidence. - -### Measurement details and reproducibility - -The table below records every current token/context efficiency measurement and its counting -boundary. - -| What is counted | Comparison | Measured reduction | Quality held constant | -|---|---|---|---| -| Cumulative reader context across a 1,986-question LoCoMo diagnostic | Full-history replay: **49,915,394** tokens → Engraphis: **891,857** tokens | **49,023,537 fewer context tokens** (**98.2133% lower**) | Focused retrieval used far less context; uncapped full history retained higher retrieval recall | -| Retrieved top-5 memory content, averaged per question | Whole documents: **808.8** tokens → structure-aware chunks: **218.4** tokens | **590.4 fewer tokens per question** (**73.0% lower**, about **3.7× smaller**) | Recall@5 **1.000** in both modes across 6 documents and 18 questions | -| Smallest returned memory that contains the reference evidence | Whole documents: **162.2** tokens → chunks: **42.4** tokens | **119.8 fewer tokens to evidence** (**73.9% lower**, about **3.8× smaller**) | The same 18 questions had a returned evidence-holding memory in both modes | -| Serialized MCP recall response across 260 timed CodeMem recalls | Full result: **17,172** `engraphis.regex.v1` tokens → compact result: **7,663** tokens | **9,509 response tokens avoided** (**55.38% lower**) | Recall@5, hit@5, and answer-token recall all **1.000** | -| Repeated-memory consolidation fixture | 12 related episodic memories: **230** tokens → one digest: **120** tokens | **110 tokens removed from the active digest** (**47.8% lower**) | Original memories remain available for provenance and audit | -| Packed prompt-context usage in the same CodeMem performance fixture | Hard budget: **1,500** tokens; observed mean: **87.73**; observed maximum: **106** | A hard cap prevents a recall from exceeding its configured context budget | This is usage accounting, not a before/after savings comparison | - -The compact MCP response avoids duplicating full memory bodies when the packed context and source -list are enough. That can reduce what an agent must inspect or pass onward, but the fixtures do -**not** measure model-provider charges, end-to-end task time, or customer cost savings. - -The measures are deliberately separate and **must not be added together**: chunking counts the -content of retrieved memory records before `ContextPacker`, whereas compact recall counts the -serialized MCP response returned to a client. “Tokens to evidence” is the size of the smallest -retrieved memory record holding the reference evidence; it is not latency or end-to-end answer -accuracy. Chunking creates more focused stored records (24 chunks rather than 6 whole-document -memories in this fixture), so this is a context-efficiency result, not a storage-reduction claim. - -Reproduce the quality and token/context measurements without a network connection or API key: - -```bash -python -m eval.harness --dataset eval/datasets/codemem.jsonl --k 5 -python -m eval.grounded -python -m eval.chunking_eval -python -m eval.performance --dataset eval/datasets/codemem.jsonl --k 5 --iterations 10 --json -``` - -These are small deterministic correctness and efficiency fixtures, not official LoCoMo / -LongMemEval QA scores or a third-party leaderboard result. Compact-response counts use the exact -`engraphis.regex.v1` counter; the chunking evaluation uses its documented deterministic -normalized-character estimator. Chunking measures retrieved memory content, while compact recall -measures serialized MCP response size. See [`BENCHMARKS.md`](BENCHMARKS.md) for definitions, -limitations, canonical external-evaluation requirements, and the no-unsupported-claims policy. - -
- ---- - ## Install ```bash diff --git a/eval/benchmark.py b/eval/benchmark.py index dd34ce90..91297fe7 100644 --- a/eval/benchmark.py +++ b/eval/benchmark.py @@ -256,7 +256,9 @@ def _redact_url(value: str) -> str: try: parsed = urlsplit(value) except ValueError: - return value + # An invalid authority can still contain credentials. Without a trustworthy parse, + # preserve neither the authority nor the rest of the URL in public evidence. + return "" if not parsed.scheme or not parsed.netloc: return value diff --git a/eval/longmemeval_v2_evidence.py b/eval/longmemeval_v2_evidence.py index be9d35ff..8a5ff01a 100644 --- a/eval/longmemeval_v2_evidence.py +++ b/eval/longmemeval_v2_evidence.py @@ -171,6 +171,10 @@ def build_evidence_report( per_question = Path(per_question_path) if re.fullmatch(r"[0-9a-f]{40}", reader_revision) is None: raise ValueError("reader_revision must be an immutable lowercase 40-character commit") + if bool(evaluator_model) != bool(evaluator_revision): + raise ValueError("evaluator_model and evaluator_revision must be used together") + if evaluator_revision and re.fullmatch(r"[0-9a-f]{40}", evaluator_revision) is None: + raise ValueError("evaluator_revision must be an immutable lowercase 40-character commit") source_paths = [ per_question, Path(haystack_path), diff --git a/tests/test_benchmark_evidence.py b/tests/test_benchmark_evidence.py index 69f06a6c..dfc7a79f 100644 --- a/tests/test_benchmark_evidence.py +++ b/tests/test_benchmark_evidence.py @@ -442,6 +442,10 @@ def test_command_provenance_redacts_userinfo_when_a_url_port_is_malformed(): ] +def test_command_provenance_fails_closed_when_url_splitting_rejects_userinfo(): + assert redact_command(["https://user:password@[invalid/path"]) == [""] + + def test_canonical_profile_validator_and_immutable_artifact_writer(tmp_path): dataset = tmp_path / "fixture.jsonl" dataset.write_text('{"id":"one"}\n', encoding="utf-8") diff --git a/tests/test_longmemeval_v2_evidence.py b/tests/test_longmemeval_v2_evidence.py index ff896680..09e30698 100644 --- a/tests/test_longmemeval_v2_evidence.py +++ b/tests/test_longmemeval_v2_evidence.py @@ -106,6 +106,28 @@ def test_official_v2_evidence_export_rejects_unpinned_reader_metadata(tmp_path): ) +@pytest.mark.parametrize( + ("evaluator_model", "evaluator_revision", "message"), + [ + ("example/evaluator", None, "must be used together"), + ("example/evaluator", "main", "immutable lowercase 40-character commit"), + ], +) +def test_official_v2_evidence_export_requires_a_pinned_evaluator( + evaluator_model, evaluator_revision, message, +): + with pytest.raises(ValueError, match=message): + build_evidence_report( + per_question_path="unused.jsonl", + questions_path="unused-questions.json", + haystack_path="unused-haystack.json", + trajectories_path="unused-trajectories.json", + memory_config_path="unused-memory.json", + evaluator_model=evaluator_model, + evaluator_revision=evaluator_revision, + ) + + def test_official_v2_evidence_export_rejects_malformed_measured_fields(tmp_path): source_paths = [ _write_json(tmp_path / name, {} if name != "questions.json" else [])