Skip to content

Image-strip fallback makes the model act as if it saw an image it never received (and leaks the file path) #4345

Description

@BearMett

Summary

nanobot has a provider-agnostic fallback (LLMProvider._strip_image_content, added in
c4628038): when a model returns a non-transient error on image_url input, the image
blocks are stripped and the turn is retried as text. This fallback is correct and
desirable — but the text it substitutes is [image: <path>] (or [image omitted]),
which reads like a live, available image.

So when an image is not actually delivered to the model, the model:

  1. acts as if it saw the image and describes contents that aren't there,
  2. tries to read_file the leaked path, and
  3. is exposed to an internal server file path it never needs.

This is core behavior in nanobot/providers/base.py, inherited by every provider — it
is not specific to any one model or backend.

Observed behavior

A user sent the plain black-and-white line drawing below (people talking on top,
people doing crafts on the bottom — no color, no characters) and asked whether the bot
could see it. The transcript:

User: (attaches the image) Can you see this image?

Bot: Yes, I can see it. The image shows a yellow character that looks like a
KakaoFriends character on a black background, with text below it reading "파프리카를
먹으면 화가 나는데…" ("when I eat paprika I get angry…").

User: That's a lie.

Nothing the bot described — the black background, the yellow character, the text — was
in the picture. The bot never received the image, but instead of saying so it fabricated
a completely different one.

The actual image that was sent:

Image

Observed with a text-only model behind an OpenAI-compatible endpoint that rejects
image_url. The model/backend is not the cause — it correctly rejects images it
cannot process; the misleading placeholder is produced by nanobot's fallback. Any
non-vision / gateway / unknown model that rejects image input triggers the same path.

Root cause

nanobot/providers/base.py, _strip_image_content / _strip_image_content_inplace:

path = (b.get("_meta") or {}).get("path", "")
placeholder = image_placeholder_text(path, empty="[image omitted]")

[image: <path>] is a neutral "an image was here" marker. It does not signal that
the image is gone and unviewable, so a model that just lost the pixels treats the
marker as a live reference and fills in the blank. The embedded path additionally
invites read_file attempts and exposes a server-internal path.

Reproduction (model-agnostic)

  1. Configure any model/provider that returns a non-transient error on image_url
    input (any non-vision, gateway, or unknown model).
  2. Send a message containing an image.
  3. Observe: the retry replaces the image with [image: <path>], and the model
    describes a non-existent image and/or attempts to read_file the path.

Scope note (why this is its own fix)

This is the failure mode that fires when an image must be dropped. It is distinct
from, and complementary to:

Proposed fix

In the error-retry strip path only, replace the path-bearing placeholder with a
neutral, state-based marker that tells the model the image is gone and unviewable,
without asserting a cause or exposing a path:

[image omitted and cannot be viewed]

Leave image_placeholder_text and its history-serialization callers
(agent/loop.py, session/manager.py) unchanged — those represent images the model
did see in an earlier turn, where "cannot be viewed" would be inaccurate.

PR to follow.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions