fix(ingestion): ingest non-existent absolute-looking strings as text [SDK-234] (gh #3887, follow-up to #3892)#4155
Merged
Conversation
Contributor
|
Hello @rshkarin, thank you for submitting a PR! We will respond as soon as possible. |
rshkarin
force-pushed
the
feature/sdk-234-windows-posix-path-ingestion
branch
4 times, most recently
from
July 22, 2026 08:58
9303e8e to
cc6db4e
Compare
rshkarin
force-pushed
the
feature/sdk-234-windows-posix-path-ingestion
branch
from
July 23, 2026 08:23
cc6db4e to
1b4595c
Compare
dexters1
added a commit
that referenced
this pull request
Jul 23, 2026
…r [SDK-307] (#4193) ## What & why The **Code Quality** CI job (`uv run ty check .`) is currently **red on every PR**, including already-merged ones — the check is broken repo-wide, not by any individual PR. **Root cause:** `cognee/infrastructure/loaders/core/image_loader.py` imports Pillow (`PIL`) for its optional EXIF-metadata and perceptual-hash features. The imports are guarded at runtime with `try/except ImportError` (graceful degradation), so Pillow is genuinely optional — but it is **not** installed by the Code Quality job (which runs only `uv sync --extra dev`; Pillow reaches the env only transitively via the `unstructured` extra). Because `[tool.ty.src].include` covers `cognee/infrastructure/loaders`, ty type-checks that file and emits **5 `unresolved-import` diagnostics** for `PIL` / `PIL.ExifTags`. **Evidence it's pre-existing and repo-wide:** Code Quality = FAILURE on #4155, #4156, #4134 and the already-merged #4153; `image_loader.py` and the ty config are byte-identical across the last ~34 dev commits. ## Change Add an inline `# ty: ignore[unresolved-import]` at each of the 5 optional `PIL` import sites — matching the existing convention in the repo (e.g. `litellm_instructor/.../azure_openai/adapter.py`). These imports are intentionally optional and already runtime-guarded, so suppressing the *static* unresolved-import there is the correct semantics. **No dependency or lockfile change**, and the `try/except` guards are untouched. Alternative considered: add Pillow to the `dev` extra so ty can actually resolve it (real type coverage of the PIL path) — heavier (lockfile change) and inconsistent with how other optional imports are handled; left as a possible later improvement. ## Verification (local, ty resolving against a full dev env with Pillow absent) - `ty check .`: **5 diagnostics → 0** ("All checks passed!"). - `ruff check` and `ruff format --check`: clean. Linear: **SDK-307**. Unblocks the Code Quality gate for all open PRs. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
rshkarin
force-pushed
the
feature/sdk-234-windows-posix-path-ingestion
branch
from
July 23, 2026 09:01
1b4595c to
fdc6897
Compare
rshkarin
marked this pull request as ready for review
July 23, 2026 09:01
rshkarin
force-pushed
the
feature/sdk-234-windows-posix-path-ingestion
branch
from
July 23, 2026 10:16
fdc6897 to
638aeda
Compare
…[SDK-234] Follow-up to #3892 (gh #3887). save_data_item_to_storage converted any absolute-looking string to a file:// URI even when no such file existed, so a "/"-prefixed text note (e.g. "/remember to call Bob") became a broken file:// URI on POSIX — and, before #3892, crashed on Windows. Gate the absolute-path branch on the file actually existing (reusing the already-sanitized abs_path), mirroring the relative-path branch: a "/"-prefixed string that is not an existing file now falls through to text ingestion on every platform. The ACCEPT_LOCAL_FILE_PATH rejection of existing local files is preserved. Mirror the same rule in the dry-run estimator (_path_candidate), which predicts ingestion routing: a non-existent absolute path is raw text, and the gate rejects only existing absolute files. The regression tests now run on the Linux/mac CI (not only the Windows matrix): non-existent absolute path -> text; Windows-style normalization -> text without raising (the module's os reference is swapped, pathlib left untouched); and the existing-file gate-off rejection. A genuine drive-anchored Windows path stays a Windows-only test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up hardening from an adversarial verification pass. Two test-only additions to lock in behavior the fix changed: - existing directory (absolute) -> text ingestion (is_file() is False, so it no longer becomes a file:// URI). Documents the deliberate divergence from the dry-run estimator, which rejects directories with a clearer message. - non-existent absolute path with ACCEPT_LOCAL_FILE_PATH off -> still text (the gate rejects only existing local files), completing mirror-parity with the estimator's gate-off assertion. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
rshkarin
force-pushed
the
feature/sdk-234-windows-posix-path-ingestion
branch
from
July 23, 2026 13:23
638aeda to
f3b33bb
Compare
Vasilije1990
approved these changes
Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Follow-up to #3892 (now merged to
dev), closing the remaining half of #3887.#3892 stops the Windows
ValueErrorcrash by guarding the absolute-path branch ofsave_data_item_to_storagewithis_absolute(). This PR fixes the other half of the report: a/-prefixed string that is not an actual file (e.g. a text note like/remember to call Bob about the meeting) should be ingested as text, not turned into afile://URI pointing at a file that does not exist.The absolute-path branch converted unconditionally, whereas the relative-path branch right below it already required the file to exist (
abs_path.is_file()). I made the two consistent by reusing that same, already-computedabs_path: an absolute-looking string is only turned into afile://URI when it points at an existing file — otherwise it falls through to text ingestion, on every platform. TheACCEPT_LOCAL_FILE_PATH=falserejection of existing local files is unchanged.While tracing the callers I found the dry-run estimator (
cognee/modules/cognify/estimator.py) deliberately mirrors this routing, and it was raising "file does not exist" for missing absolute paths. Since a real run now treats those as text, I updated_path_candidateto match (missing path → raw text; the gate rejects only existing absolute files) and adjusted its two tests.Behavior note: because the branch now keys on "existing file" (
is_file()), a handful of absolute-looking strings that are not regular files also route to text instead of afile://URI — an existing directory path, a broken symlink, and pathological paths that error duringresolve()(embedded null / over-length). This is intended and safe: none of those are ingestible files, and directories are normally expanded upstream byresolve_data_directoriesbefore reaching here. It also repairs a latent downstream break — before this change a/-prefixed note produced a brokenfile://URI thatget_data_file_pathturned into a non-existent path and the loader then failed on.Acceptance Criteria
save_data_item_to_storage("/remember to call Bob about the meeting")is ingested as text — nofile://URI, no crash — on Windows, macOS and Linux.file://URI; withACCEPT_LOCAL_FILE_PATH=falseit still raisesIngestionError.ruff format+ruff checkclean.Type of Change
Pre-submission Checklist
CONTRIBUTING.md)DCO Affirmation
I affirm that all code in every commit of this pull request conforms to the terms of the Topoteretes Developer Certificate of Origin.