invalid escape sequence DeprecationWarning from template syntax itself

[asottile]

mcve

(this is extracted from a larger example which was rendering actual .j2 files -- however this was the shortest example I could come up with)

from jinja2 import Template

print(Template(r'{{ "\d" }}').render())

Expected Behavior

I'm unclear on the expected behaviour, there's kinda 3 options that I see as reasonable (unranked since I don't really have an opinion here)

• warning: produce a warning, but with better indication on the user's fix (especially: point at the actual filename where the problem is)
• error: make invalid escape sequences in templates an error
• (silence) accept that string tokens in jinja source can have escape sequences that do not comply with python's rules

Actual Behavior

$ python3 -Wonce t.py
/tmp/x/venv/lib/python3.6/site-packages/jinja2/lexer.py:606: DeprecationWarning: invalid escape sequence '\d'
  .decode("unicode-escape")
\d

Full Traceback

This can be converted into an error with -Werror

$ python3 -Werror t.py
DeprecationWarning: invalid escape sequence '\d'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/tmp/x/venv/lib/python3.6/site-packages/jinja2/lexer.py", line 606, in wrap
    .decode("unicode-escape")
DeprecationWarning: decoding with 'unicode-escape' codec failed (DeprecationWarning: invalid escape sequence '\d')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "t.py", line 3, in <module>
    print(Template(r'{{ "\d" }}').render())
  File "/tmp/x/venv/lib/python3.6/site-packages/jinja2/environment.py", line 986, in __new__
    return env.from_string(source, template_class=cls)
  File "/tmp/x/venv/lib/python3.6/site-packages/jinja2/environment.py", line 896, in from_string
    return cls.from_code(self, self.compile(source), globals, None)
  File "/tmp/x/venv/lib/python3.6/site-packages/jinja2/environment.py", line 625, in compile
    self.handle_exception(source=source_hint)
  File "/tmp/x/venv/lib/python3.6/site-packages/jinja2/environment.py", line 787, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<unknown>", line 1, in template
jinja2.exceptions.TemplateSyntaxError: invalid escape sequence '\d')

Your Environment

$ python --version --version
Python 3.6.9 (default, Nov  7 2019, 10:44:02) 
[GCC 8.3.0]
$ pip freeze --all
Jinja2==3.0.0a1
MarkupSafe==1.1.1
pip==20.0.2
setuptools==45.2.0
wheel==0.34.2

---

The "correct" fix for those hitting the same issue:

making a raw string is not valid syntax in jinja so that is not an option:

$ diff -u t.py t.py.new
--- t.py	2020-02-18 09:53:02.892139615 -0800
+++ t.py.new	2020-02-18 09:52:57.860139615 -0800
@@ -1,3 +1,3 @@
 from jinja2 import Template
 
-print(Template(r'{{ "\d" }}').render())
+print(Template(r'{{ r"\d" }}').render())

$ python3 t.py.new
...
jinja2.exceptions.TemplateSyntaxError: expected token 'end of print statement', got 'string'

the "fix" I needed was to manually escape the backslashes:

$ diff -u t.py t.py.new
--- t.py	2020-02-18 09:53:02.892139615 -0800
+++ t.py.new	2020-02-18 09:53:57.068139615 -0800
@@ -1,3 +1,3 @@
 from jinja2 import Template
 
-print(Template(r'{{ "\d" }}').render())
+print(Template(r'{{ "\\d" }}').render())

$ python3 t.py.new
\d

[dtantsur]

As a data point: Ansible works around this behaviour by magically inserting missing slashes: https://github.com/ansible/ansible/blob/42bc03f0f5740f2340fcdbe75557920552622ac3/lib/ansible/template/__init__.py#L130