fix: hybrid-memory cron maintenance outcomes and exit ledger validation by Claude · Pull Request #1204 · markus-lassfolk/openclaw-hybrid-memory

Claude · 2026-05-08T05:53:26Z

Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a plan and make progress.

This section details on the original issue you should resolve

<issue_title>Bug: hybrid-memory cron jobs report OK despite failed or partial maintenance</issue_title>
<issue_description>## Summary

Hybrid-memory maintenance cron runs can be recorded as status: ok / lastRunStatus: ok even when the actual maintenance task failed, exited early, or only completed the first step. This hides real maintenance failures from cron health, consecutiveErrors, and failure alerts.

This is dangerous because the scheduler/top-level state says the system is healthy while the logs and run summary say maintenance did not complete.

Concrete evidence from 2026-05-08

Host: Maeve / /home/markus/.openclaw
Plugin version in logs: openclaw-hybrid-memory v2026.5.70, memory-manager 3.0
OpenClaw version in logs: OpenClaw 2026.5.4 (325df3e)

1. hybrid-mem:nightly-distill reported OK despite failing before steps 2-5

Cron run state reported:
{
  "jobId": "hybrid-mem:nightly-distill",
  "action": "finished",
  "status": "ok",
  "summary": "The nightly memory sweep cron job failed at checking if distill is enabled due to a shell syntax error: \"bad substitution\" ... This error caused the script to exit early and not continue with steps 2-5.",
  "runAtMs": 1778205600008,
  "durationMs": 46978
}
The job state also showed:
"lastRunStatus": "ok",
"lastStatus": "ok",
"consecutiveErrors": 0
But the summary explicitly says it failed and exited early.

File-level evidence:
/home/markus/.openclaw/logs/cron-hybrid-mem/20260508/nightly-distill-20260508T020036Z-14599.exit.txt
2026-05-08T02:00:40Z prune exit=0
Only prune is recorded. Required steps that were supposed to run after that are missing:

distill --days 1 --verbose

extract-daily --verbose

resolve-contradictions

enrich-entities --limit 200 --verbose

Guard evidence confirms it did not record a fresh successful run:
/home/markus/.openclaw/cron/guard/nightly-memory-sweep.ms = 1778119820161
= 2026-05-07T02:10:20.161Z
The failed 2026-05-08 run did not update the guard, which is correct, but cron still considered the run OK.

2. hybrid-mem:nightly-dream-cycle reported OK despite unknown command

Cron run state reported:
{
  "jobId": "hybrid-mem:nightly-dream-cycle",
  "action": "finished",
  "status": "ok",
  "summary": "I attempted to poll the process for completion but did not provide the session ID...",
  "runAtMs": 1778208300006,
  "durationMs": 34708
}
But the job log contains:
/home/markus/.openclaw/logs/cron-hybrid-mem/20260508/dream-cycle-20260508T024515Z-9831.log:66:error: unknown command 'consolidate-episodes'
The exit ledger contains only:
/home/markus/.openclaw/logs/cron-hybrid-mem/20260508/dream-cycle-20260508T024515Z-9831.exit.txt
2026-05-08T02:45:19Z prune exit=0
So again only prune is recorded, while the later intended dream-cycle/consolidation/reflection work did not complete.

Guard evidence also shows no fresh successful dream cycle was recorded:
/home/markus/.openclaw/cron/guard/nightly-dream-cycle.ms = 1778035529225
= 2026-05-06T02:45:29.225Z
3. self-correction-analysis skipped due to guard but status semantics are muddy

The 2026-05-08 self-correction-analysis run reports ok because it skipped due to guard:
{
  "jobId": "hybrid-mem:self-correction-analysis",
  "status": "ok",
  "summary": "The guard check indicates that the self-correction analysis was run within the last 20 hours. Therefore, it will be skipped..."
}
Skipping due to guard may be acceptable as skipped, but it should not be indistinguishable from a successful maintenance run when assessing freshness. hybrid-mem stats later reported stale/overdue jobs, which suggests there is already disagreement between scheduler state and plugin health interpretation.

Pattern seen in prior runs

This is not a one-off. Existing cron/runs/*.jsonl history contains many runs where status: ok wraps summaries such as:

Nightly memory sweep ran, but not cleanly.

distill --days 3: failed / aborted

timed out at batch ...

Guard file was not updated because the full run did not succeed.

nightly-dream-cycle failed.

partial success, hit outer timeout

Patterns extracted: did not complete

Rules generated: did not run

Those are all operational failures or partial failures, but the scheduler records them as successful agent turns because the agent produced a textual summary rather than throwing a hard process error.

Expected behaviour

Maintenance jobs should fail closed:

If a required step fails, is missing from the exit ledger, returns non-zero, hits an unknown command, times out, or does not update the success guard, the cron run should be marked failed or at least partial/degraded.

lastRunStatus, lastStatus, and consecutiveErrors should reflect maintenance outcome, not merely “the agent turn produced a response”.

Failure alerts should fire for required-step failures.

A guard-skip should be reported as skipped, not ok, and should not reset freshness incorrectly.

The final status should be derived from structured step results, not from free-text agent summaries.

Suggested fix

Add a structured maintenance runner/contract for plugin-managed cron jobs:

Each maintenance cron should write an exit ledger with all expected required steps.

The wrapper should validate that all expected required steps are present and exit=0.

If any required step is absent, non-zero, timed out, or the command text reports unknown command, mark the job failed.

Only write the guard timestamp after all required steps complete successfully.

Return/emit machine-readable outcome, for example:
{
  "maintenanceStatus": "success|skipped|partial|failed",
  "requiredSteps": [
    {"name":"prune","exit":0},
    {"name":"distill","exit":1,"error":"bad substitution"}
  ],
  "guardUpdated": false,
  "logPath": "...",
  "exitPath": "..."
}
Then map:

success -> cron ok

skipped -> cron skipped

partial / failed -> cron error or equivalent degraded status

If OpenClaw core only supports ok/error/skipped, plugin-managed jobs should force an agent/process non-zero outcome or raise an exception when maintenanceStatus != success|skipped.

Related issues noticed while auditing

A. Stale command in dream-cycle cron payload

The 2026-05-08 dream-cycle log shows:
error: unknown command 'consolidate-episodes'
This looks like the cron payload/runbook references a CLI subcommand that no longer exists or was renamed. The managed cron generation should be updated, and ideally hybrid-mem verify should detect stale managed cron payloads that call unknown subcommands.

B. Shell/config checks are brittle

The 2026-05-08 nightly-distill run failed with:
bad substitution
The summary says this happened while checking plugins.entries["openclaw-hybrid-memory"].config.distill.enabled via shell syntax. Config checks in generated cron prompts/scripts should use a robust mechanism (node, jq, or plugin CLI config output), not Bash parameter expansion against dotted/quoted config paths.

C. Guard and plugin thresholds disagree

Several self-correction and distill runs show confusion between a 20h cron guard and an internal 23h plugin threshold. This leads to runs that pass one guard but then self-skip internally, sometimes still updating cron guard. These should be aligned or represented clearly as skipped.

D. Partial dream-cycle runs sometimes update guard

Some historical summaries say reflection/rules timed out or were killed, but guard was still written and the cycle counted as complete. Guard updates should be strictly tied to all required stages finishing, or the stages should be explicitly classified as optional.

Impact

Operators see green cron state while memory maintenance is stale or incomplete.

Failure alerts do not fire.

consecutiveErrors remains 0 despite repeated failed/partial maintenance.

Nightly distill/dream health can silently degrade.

Troubleshooting becomes much harder because the source of truth is split between cron state, free-text summaries, exit ledgers, guard files, and hybrid-mem stats.

Acceptance criteria

A required-step failure in nightly-distill makes the cron run non-OK.

A dream-cycle unknown command makes the cron run non-OK.

Missing required step exit lines are treated as failure.

Guard-skip produces skipped, not ok.

hybrid-mem stats and cron state agree on freshness/health.

Managed cron payloads no longer reference consolidate-episodes if that command is invalid.

Generated config checks no longer use brittle Bash substitutions for nested plugin config.
</issue_description>

<agent_instructions>Look at adding handling to all the cron jobs and tasks. </agent_instructions>

Comments on the Issue (you are @claude[agent] in this section)

github-actions · 2026-05-08T05:55:55Z

Changed Files Summary

Source (5)

extensions/memory-hybrid/cli/cmd-install.ts
extensions/memory-hybrid/cli/commands/manage/register-validate-cron-exit.ts
extensions/memory-hybrid/cli/commands/register-manage-commands.ts
extensions/memory-hybrid/services/cron-exit-validator.ts
extensions/memory-hybrid/services/cron-job-bash-harness.ts

Tests (1)

extensions/memory-hybrid/tests/cron-exit-validator.test.ts

Docs (2)

CHANGELOG.md
extensions/memory-hybrid/docs/cron-exit-validation.md

Total: 8 files changed

github-actions · 2026-05-08T05:55:56Z

Forge Feedback Loop

PR: fix: hybrid-memory cron maintenance outcomes and exit ledger validation #1204
Branch: claude/fix-hybrid-memory-cron-jobs
Head SHA: 4e81e3d1f184835d02429eb591631d80997893a1
Failed checks: 0
New PR comments since latest push: 0
New review summaries since latest push: 0
Unresolved review threads: 3
Status: dispatching Forge
Reasons: review-feedback
Dispatch: queued forge-pr-remediation.requested for Forge/Codex gpt-5.4-pro.

Copilot

Copilot wasn't able to review any files in this pull request.

- Create cron-exit-validator.ts with structured validation - Add validate-cron-exit CLI command for internal use - Update cron job bash harness with validation instructions - Add comprehensive tests for exit ledger parsing and validation - Register new validation command in CLI Agent-Logs-Url: https://github.com/markus-lassfolk/openclaw-hybrid-memory/sessions/cd60b52e-d486-4166-b77f-63a3040389a2 Co-authored-by: markus-lassfolk <3661143+markus-lassfolk@users.noreply.github.com>

- Update existing cron job messages when normalizeExisting=true - Remove obsolete command references (e.g., consolidate-episodes) - Convert tests to vitest format - All 14 tests passing Agent-Logs-Url: https://github.com/markus-lassfolk/openclaw-hybrid-memory/sessions/cd60b52e-d486-4166-b77f-63a3040389a2 Co-authored-by: markus-lassfolk <3661143+markus-lassfolk@users.noreply.github.com>

- Document validation behavior and expected outcomes - Explain migration path for existing installations - Detail config-based skipping mechanism - List acceptance criteria and testing coverage Agent-Logs-Url: https://github.com/markus-lassfolk/openclaw-hybrid-memory/sessions/cd60b52e-d486-4166-b77f-63a3040389a2 Co-authored-by: markus-lassfolk <3661143+markus-lassfolk@users.noreply.github.com>

…gelog - Pin CI/security to Node 22.16 (node:sqlite + FTS5, nodejs/node#57621) - engines, .nvmrc, docs: require >=22.16.0 - Tests: storageGrowth nulls vs expect.anything(); EventLog CHECK relaxed - CHANGELOG [Unreleased]: cron exit validation (#1203) + Node bump - Biome-format cron exit validator / harness / install paths Co-authored-by: Cursor <cursoragent@cursor.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: da95333ac5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

… error) Co-authored-by: Cursor <cursoragent@cursor.com>

Empty or step-less HM_EXIT cannot distinguish guard skip from abort before hm_step; report failure so validate-cron-exit surfaces the error. Co-authored-by: chatgpt-codex-connector <noreply@openai.com> Co-authored-by: Cursor <cursoragent@cursor.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bfa27f2657

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…sting Apply overrides to payload.message for agentTurn jobs and skip canonical message replacement when a per-job override is provided. Co-authored-by: Cursor <cursoragent@cursor.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c2ed444fdb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-05-08T11:30:19Z

+    "",
+    "VALIDATION & GUARD UPDATE (Issue: cron jobs report OK despite failures)",
+    `After the bash script completes, validate that ALL required steps [${requiredStepsList}] appear in HM_EXIT with exit=0.`,
+    "- If ANY required step is missing from HM_EXIT, has exit≠0, or the log contains 'unknown command', this job has FAILED.",


Allow full-job config skips in cron validation flow

The new generic validation rule marks any missing required step as FAILED, which conflicts with several job preambles that explicitly instruct a full skip when a feature is disabled (for example self-correction-analysis and weekly-reflection in cmd-install.ts). In that disabled-feature scenario, no hm_step lines are expected in HM_EXIT, so the current wording forces a failed outcome instead of the intended skipped outcome and can incorrectly increment error counts for legitimate guard/config skips.

Useful? React with 👍 / 👎.

…gated skip Co-authored-by: Cursor <cursoragent@cursor.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4e81e3d1f1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-05-08T13:18:08Z

+    `After the bash script completes, validate that ALL required steps [${requiredStepsList}] appear in HM_EXIT with exit=0.`,
+    "- If ANY required step is missing from HM_EXIT, has exit≠0, or the log contains 'unknown command', this job has FAILED.",
+    "- If a step is replaced with a config-skip variant (e.g., 'distill-skipped' exit=0 when distill.enabled is false), that counts as present.",
+    "- Only after ALL required steps are validated successful: perform the GUARD CHECK timestamp write.",
+    "- If validation fails, do NOT update the guard file.",


Enforce exit-ledger validation in the execution path

The updated cron template only describes validation after the bash block, but it never runs openclaw hybrid-mem validate-cron-exit (or any other command that can fail the turn). When the model omits or misreports that manual validation step, cron can still mark the run successful even if required HM_EXIT steps are missing/failed, which recreates the false-green behavior this fix is meant to prevent.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-05-08T13:18:08Z

+    if (text.length === 0) return false;
+    if (/\bjob was skipped\b/.test(text)) return true;
+    if (/\bskip(ped|ping)?\b.*\bhybrid[- ]memory\b.*\bconfig\b/.test(text)) return true;
+    if (/\bself[- ]correction\b.*\b(disabled|skipped)\b/.test(text)) return true;


Match camelCase feature names in skip-log heuristics

The skip detector only matches self-correction with a hyphen/space, but several cron preambles use camelCase config keys such as selfCorrection.enabled and nightlyCycle.enabled (cmd-install.ts). If logs follow those key names (for example, "selfCorrection.enabled is false, skipping"), this regex does not match, so an intentionally skipped full job with an empty ledger is incorrectly marked failed.

Useful? React with 👍 / 👎.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e427f30db0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-05-08T16:50:20Z

+
+    for (const line of lines) {
+      // Match patterns like: "error: unknown command 'consolidate-episodes'"
+      const match = line.match(/(?:error|Error):\s*unknown command\s+['"]([^'"]+)['"]/);


Limit unknown-command scan to failing step output

checkForUnknownCommands fails the whole run whenever any HM_LOG line contains error: unknown command ..., but this scans all log text, not just the executed command failure stream. The maintenance-log-analyzer cron job (defined in cmd-install.ts) intentionally prints excerpts from prior maintenance logs, so a successful analyzer run can legitimately include historical unknown command strings and be misclassified as failed. This creates false failures and can inflate cron error counts even when required steps exited 0.

Useful? React with 👍 / 👎.

Initial plan

70bac0d

Claude AI assigned Claude and markus-lassfolk May 8, 2026

Claude AI linked an issue May 8, 2026 that may be closed by this pull request

Bug: hybrid-memory cron jobs report OK despite failed or partial maintenance #1203

Closed

Claude started work on behalf of markus-lassfolk May 8, 2026 05:53 View session

markus-lassfolk requested a review from Copilot May 8, 2026 05:55

github-actions Bot added the size/XS label May 8, 2026

Copilot AI reviewed May 8, 2026

View reviewed changes

Claude AI and others added 3 commits May 8, 2026 06:00

Claude AI requested a review from markus-lassfolk May 8, 2026 06:06

Claude finished work on behalf of markus-lassfolk May 8, 2026 06:06

markus-lassfolk approved these changes May 8, 2026

View reviewed changes

github-actions Bot added extensions size/XL and removed size/XS labels May 8, 2026

markus-lassfolk and others added 2 commits May 8, 2026 10:48

Merge branch 'main' into claude/fix-hybrid-memory-cron-jobs

3141a0c

markus-lassfolk changed the title ~~[WIP] Fix hybrid-memory cron jobs reporting wrong status~~ fix: hybrid-memory cron maintenance outcomes and exit ledger validation May 8, 2026

markus-lassfolk marked this pull request as ready for review May 8, 2026 09:15

github-actions Bot added documentation Improvements or additions to documentation github-actions packages labels May 8, 2026

chatgpt-codex-connector Bot reviewed May 8, 2026

View reviewed changes

Comment thread extensions/memory-hybrid/services/cron-exit-validator.ts Outdated

Ralph and others added 2 commits May 8, 2026 11:21

fix(memory-hybrid): use Number.parseInt in cron-exit-validator (Biome…

61eeead

… error) Co-authored-by: Cursor <cursoragent@cursor.com>

markus-lassfolk enabled auto-merge (squash) May 8, 2026 10:08

chatgpt-codex-connector Bot reviewed May 8, 2026

View reviewed changes

Comment thread extensions/memory-hybrid/cli/cmd-install.ts Outdated

fix(memory-hybrid): respect messageOverrides during cron normalizeExi…

c2ed444

…sting Apply overrides to payload.message for agentTurn jobs and skip canonical message replacement when a per-job override is provided. Co-authored-by: Cursor <cursoragent@cursor.com>

chatgpt-codex-connector Bot reviewed May 8, 2026

View reviewed changes

fix(cron): treat empty exit ledger as skipped when log shows feature-…

4e81e3d

…gated skip Co-authored-by: Cursor <cursoragent@cursor.com>

chatgpt-codex-connector Bot reviewed May 8, 2026

View reviewed changes

merge main into cron-jobs PR

e427f30

markus-lassfolk merged commit a95e1c0 into main May 8, 2026
10 checks passed

markus-lassfolk deleted the claude/fix-hybrid-memory-cron-jobs branch May 8, 2026 16:45

chatgpt-codex-connector Bot reviewed May 8, 2026

View reviewed changes

Navigation Menu

fix: hybrid-memory cron maintenance outcomes and exit ledger validation - #1204

fix: hybrid-memory cron maintenance outcomes and exit ledger validation#1204
markus-lassfolk merged 11 commits into
mainfrom
claude/fix-hybrid-memory-cron-jobs

Claude AI commented May 8, 2026

Uh oh!

github-actions Bot commented May 8, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented May 8, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot May 8, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot May 8, 2026

Uh oh!

chatgpt-codex-connector Bot May 8, 2026

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot May 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Claude AI commented May 8, 2026

Concrete evidence from 2026-05-08

1. hybrid-mem:nightly-distill reported OK despite failing before steps 2-5

2. hybrid-mem:nightly-dream-cycle reported OK despite unknown command

3. self-correction-analysis skipped due to guard but status semantics are muddy

Pattern seen in prior runs

Expected behaviour

Suggested fix

Related issues noticed while auditing

A. Stale command in dream-cycle cron payload

B. Shell/config checks are brittle

C. Guard and plugin thresholds disagree

D. Partial dream-cycle runs sometimes update guard

Impact

Acceptance criteria

Comments on the Issue (you are @claude[agent] in this section)

Uh oh!

github-actions Bot commented May 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changed Files Summary

Source (5)

Tests (1)

Docs (2)

Uh oh!

github-actions Bot commented May 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Forge Feedback Loop

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot May 8, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot May 8, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot May 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot May 8, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

1. `hybrid-mem:nightly-distill` reported OK despite failing before steps 2-5

2. `hybrid-mem:nightly-dream-cycle` reported OK despite `unknown command`

3. `self-correction-analysis` skipped due to guard but status semantics are muddy

github-actions Bot commented May 8, 2026 •

edited

Loading

github-actions Bot commented May 8, 2026 •

edited

Loading