Skip to content

fix: keep jinja2 blocks intact when splitting tags from imported URLs (#1516)#4259

Open
ebarkhordar wants to merge 1 commit into
dgtlmoon:masterfrom
ebarkhordar:fix/1516-import-jinja2-url-split
Open

fix: keep jinja2 blocks intact when splitting tags from imported URLs (#1516)#4259
ebarkhordar wants to merge 1 commit into
dgtlmoon:masterfrom
ebarkhordar:fix/1516-import-jinja2-url-split

Conversation

@ebarkhordar

Copy link
Copy Markdown
Contributor

Root cause

import_url_list.run() separates each import line into URL and tags by splitting on the first space:

if ' ' in url:
    url, tags = url.split(" ", 1)

A jinja2 block in a watch URL can legitimately contain spaces, so the split cuts the block in half. Importing

https://example.com/?date={% now 'utc', '%d' %} tag-a

produces url="https://example.com/?date={%" and tags="now 'utc', '%d' %} tag-a".

The truncated URL is then rejected downstream: add_watch() calls is_safe_valid_url(), which jinja2-renders any URL containing {%, and ?date={% is a template syntax error. So add_watch() returns None and the line is counted as skipped rather than imported. Jinja2 in a watch URL is a supported feature (is_safe_valid_url() renders it deliberately, and the quick-add form has test_jinja2_time_offset_in_url_query for it), so the same URL is accepted when added through the form and mangled when imported.

The invariant

Whitespace inside a jinja2 block is part of the URL. Only whitespace outside {% %} / {{ }} can separate the URL from its tags.

The fix

split_url_and_tags() masks the jinja2 blocks (keeping the length, so offsets still map back to the line) and splits on the first space outside them. Lines without a jinja2 block take exactly the previous path.

I kept this to the jinja2 case only. A URL with a literal, un-encoded space (#1568) stays ambiguous, since the import format uses a space as the URL/tag separator, and disambiguating that is a separate decision.

How I verified it

All of this ran in the test-changedetectionio container built from this repo's Dockerfile at 5eb2638 (I had to strip the BuildKit cache mounts, no buildx on my box, dependencies are otherwise unchanged):

  • Fails on master, passes here. The new test test_import_url_with_jinja2_whitespace fails on unmodified master (Imported 0 new UUIDs, the watch is skipped) and passes with the fix.
  • pytest tests/test_import.py: 7 passed, so the existing space-separated tag tests (https://example.com tag1, other tag) still behave the same.
  • pytest tests/unit/: passed.
  • CI's lint gate (ruff check . --select E9,F63,F7,F82,INT, per test-only.yml): passes.

What I did not verify: tests/test_jinja2.py has 2 failures (test_jinja2_in_url_query, test_jinja2_time_offset_in_url_query) in my environment, but they fail identically on unmodified master with the same 500 from the preview page, so they are pre-existing here and unrelated to the importer. I did not run the playwright/selenium legs of the CI matrix, which this change does not touch.

I did not run pre-commit as a whole: its ruff --fix / ruff-format hooks reformat these two files wholesale (import reordering, class Importer(): to class Importer:, and so on) and they do the same to unmodified master, so I matched the existing file style and checked the lint that CI actually gates on instead. Happy to run the formatter over the files if you would rather have that.

This was written with AI assistance. The root cause, the fix, and every result above are mine and were reproduced and run as described.

Fixes #1516

…dgtlmoon#1516)

The URL list importer separates a line into URL and tags by splitting on the
first space. A jinja2 block in a watch URL can legitimately contain spaces, so
the split cuts the block in half: importing

    https://example.com/?date={% now 'utc', '%d' %} tag-a

gave url="https://example.com/?date={%" and tags="now 'utc', '%d' %} tag-a".
The truncated URL then fails is_safe_valid_url() (it jinja2-renders any URL
containing '{%', and the fragment is a syntax error), so add_watch() returns
None and the watch is silently reported as skipped.

Split on the first space that sits outside any {% %} / {{ }} block instead.
Whitespace inside a jinja2 block belongs to the URL; only whitespace outside
one can separate the tags. Lines with no jinja2 block behave exactly as before.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

URL Import with jinja2 variables not imported correctly

1 participant