The ABAP lexer crashed on the way nearly every ABAP file begins

    *&---------------------------------------------------------------------*
    *& Report Z_SOMETHING
    *&---------------------------------------------------------------------*
    REPORT z_something.

`_scan_line` returned early on a column-1 `*` comment — before the line that
recorded whether the line had left a literal open. The splitter reads that record
on the very next statement, off the function object.

So the FIRST file scanned in a process whose first line is a comment raised
AttributeError. scan_tree does not catch it, so one ordinary abapGit banner
failed the whole abap_sast module and the run reported no custom-code findings at
all. I found it by writing a fixture that opens with a comment, which no existing
fixture does — all nine start on a keyword.

The second defect is quieter and was the reason to fix this properly rather than
set the attribute on one more path. Once any non-comment line had set it, the
value persisted ON THE FUNCTION across comment lines and across FILES. A file
ending inside an open literal left it True, and the next file to begin with a
comment had its first statement marked `degraded` — a lexer fault attributed to
an innocent file, and one more on the `lex_degraded` count that raises a coverage
finding about mis-lexed objects.

`_scan_line` now returns the fact as its fifth value, so a path that forgets to
produce it cannot compile rather than reading a stale one, and the attribute is
gone. The comment defending the original design cited "four call sites [that]
depend on" the signature; there were two.

A comment line returns False rather than recomputing from the carried stack —
they are equivalent on every stack the comment guard can be reached with, and
False says the thing that is true: a comment opens nothing and closes nothing.

Three of the four mutations run against the new test fail it; the fourth was
provably a no-op rewrite of the same constant, not an escaped defect.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: Krishcalin

modules/abap_sast.py

@@ -572,8 +572,8 @@ def emit(text: str, masked: str, degraded: bool = False) -> None:
             while _in_amdp(stack):
                 stack.pop()
 
-        code, mask, term, stack = _scan_line(line, stack)
-        if _scan_line.unterminated_at_eol:
+        code, mask, term, stack, unterminated = _scan_line(line, stack)
+        if unterminated:
             pending_degraded = True
         raw.append(line)
 
@@ -934,8 +934,12 @@ def put(ch: str, m: Optional[str] = None, t: str = "0") -> None:
 
     # A `*` in column 1 comments the whole line — but only when we are not inside
     # a literal or template left open on a previous line, where it is text.
+    #
+    # `False` for the last element and not a carried value: a comment line opens
+    # nothing and closes nothing, so it leaves nothing unterminated. Whatever the
+    # line above left open was reported when THAT line was scanned.
     if line[:1] == "*" and mode in (_M_CODE, _M_EMB, _M_AMDP):
-        return "", "", "", stack
+        return "", "", "", stack, False
 
     i, n = 0, len(line)
     while i < n:
@@ -1071,10 +1075,8 @@ def put(ch: str, m: Optional[str] = None, t: str = "0") -> None:
         i += 1
 
     closed = _close_at_end_of_line(stack)
-    #: Set on the function so the splitter can mark the statement without
-    #: changing a return signature four call sites depend on.
-    _scan_line.unterminated_at_eol = len(closed) != len(stack)
-    return "".join(code), "".join(mask), "".join(term), closed
+    return ("".join(code), "".join(mask), "".join(term), closed,
+            len(closed) != len(stack))
 
 
 def _close_at_end_of_line(stack: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
@@ -1744,7 +1746,7 @@ def __init__(self, text: str, sanitizers: Iterable[str] = DEFAULT_SANITIZERS):
         code_lines: List[str] = []
         stack: List[Tuple[str, str]] = []
         for line in self._raw:
-            _code, mask, _term, stack = _scan_line(line, stack)
+            _code, mask, _term, stack, _unterminated = _scan_line(line, stack)
             code_lines.append(_taint_rewrite(mask.strip()))
         self._code = code_lines
         # B2/B4 — a METHOD implementation header carries no signature, so the

tests/test_abap_comment_first_line.py

@@ -0,0 +1,130 @@
+"""The ABAP lexer crashed on the way nearly every ABAP file begins.
+
+    *&---------------------------------------------------------------------*
+    *& Report Z_SOMETHING
+    *&---------------------------------------------------------------------*
+    REPORT z_something.
+
+A `*` in column 1 is a full-line comment, and `_scan_line` returned early on it
+— before the line that recorded whether the line had left a literal open. The
+splitter read that record on the very next statement. So:
+
+  * The FIRST file scanned in a process whose first line is a comment raised
+    AttributeError. `scan_tree` does not catch it, so one ordinary banner failed
+    the whole `abap_sast` module and the run reported no custom-code findings at
+    all. Every abapGit export and every SE38 program starts this way.
+
+  * Once any non-comment line had set it, the value PERSISTED on the function
+    across comment lines and across FILES. A file that ended inside an open
+    literal left it True, and the next file to begin with a comment had its first
+    statement marked `degraded` — a lexer fault attributed to an innocent file,
+    which also inflates the `lex_degraded` count that raises a coverage finding.
+
+Both came from carrying the fact on the function object rather than returning it,
+and the comment that explained the choice cited "four call sites" that would have
+to change. There were two. `_scan_line` now returns it and the attribute is gone,
+so a path that forgets to set it cannot compile rather than reading a stale one.
+"""
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+import pytest
+
+ROOT = Path(__file__).resolve().parents[1]
+if str(ROOT) not in sys.path:
+    sys.path.insert(0, str(ROOT))
+
+from modules import abap_sast as sast  # noqa: E402
+
+#: What abapGit writes above `REPORT`. The literal cause of the crash.
+BANNER = (
+    "*&---------------------------------------------------------------------*\n"
+    "*& Report Z_DEMO\n"
+    "*&---------------------------------------------------------------------*\n"
+)
+CLEAN = BANNER + "REPORT z_demo.\nWRITE 'hello'.\n"
+
+#: A file whose last line leaves a text literal open. Genuinely degraded, and the
+#: source of the state that used to leak into the next file.
+UNTERMINATED = "REPORT z_a.\nDATA(x) = 'never closed\n"
+
+
+def test_a_file_that_opens_with_a_comment_does_not_raise():
+    """The crash, reproduced at its simplest.
+
+    Called first in this module so it runs against the same cold state a real
+    scan starts from — the defect was invisible to any test that had already
+    lexed a line of code."""
+    assert sast.split_statements(CLEAN), "the banner-first file produced nothing"
+
+
+def test_the_fact_is_returned_and_not_parked_on_the_function():
+    """Structural, because the behavioural tests below pass either way once the
+    attribute happens to be set. An attribute is reachable from anywhere and
+    survives the call that wrote it; a return value cannot be stale."""
+    assert not hasattr(sast._scan_line, "unterminated_at_eol"), (
+        "the lexer is carrying its per-line state on the function object again. "
+        "Return it instead — that is what made a comment line inherit the "
+        "previous line's verdict.")
+    _code, _mask, _term, _stack, unterminated = sast._scan_line("REPORT z.", [])
+    assert unterminated is False
+
+
+def test_a_comment_line_reports_nothing_unterminated():
+    """It opens nothing and closes nothing. Whatever the line above left open was
+    reported when that line was scanned."""
+    _c, _m, _t, _s, unterminated = sast._scan_line("* just a comment", [])
+    assert unterminated is False
+
+
+def test_an_unterminated_literal_is_still_degraded():
+    """The fix must not have bought its correctness by never flagging anything."""
+    flagged = [st for st in sast.split_statements(UNTERMINATED) if st.degraded]
+    assert flagged, "a line leaving a literal open is no longer marked degraded"
+    assert "never closed" in flagged[0].text
+
+
+def test_one_files_open_literal_does_not_degrade_the_next_file():
+    """The contamination, stated as the customer would meet it: file B is clean
+    ABAP and gets a lexer fault on its first statement because file A, scanned
+    earlier in the same run, ended mid-literal."""
+    sast.split_statements(UNTERMINATED)
+    after = sast.split_statements(CLEAN)
+    assert not any(st.degraded for st in after), (
+        "a clean file was marked degraded by the previous file's open literal: "
+        + repr([st.text for st in after if st.degraded]))
+
+
+def test_the_banner_is_not_counted_as_a_statement():
+    """Three comment lines and two statements. A banner that lexed as a statement
+    would give every rule three lines of prose to match against."""
+    texts = [st.text for st in sast.split_statements(CLEAN)]
+    assert texts == ["REPORT z_demo", "WRITE 'hello'"], texts
+
+
+@pytest.mark.parametrize("first", [
+    "*&---------------------------------------------------------------------*",
+    "* plain comment",
+    "*",
+    "*\"! ABAP Doc comment",
+])
+def test_every_column_one_comment_shape_opens_a_file_safely(first):
+    """`*` in column 1 is the comment marker whatever follows it, including the
+    ABAP Doc form, whose `"` would otherwise open a literal."""
+    assert sast.split_statements(first + "\nREPORT z.\nWRITE 'x'.\n")
+
+
+def test_the_scanner_reads_a_banner_first_file_end_to_end():
+    """Above the lexer: the module-level failure the crash actually produced was
+    `abap_sast` returning nothing for the whole tree."""
+    scanner = sast.AbapSourceScanner(data_flow=True)
+    findings = scanner.scan_text(
+        BANNER + "REPORT z_demo.\n"
+        "PARAMETERS p_t TYPE string.\n"
+        "SELECT * FROM (p_t) INTO TABLE @DATA(lt).\n",
+        Path("z_demo.prog.abap"))
+    assert any(f["rule_id"].startswith("ABAP-SQLI") for f in findings), (
+        "the dynamic FROM clause under a banner produced no injection finding: "
+        + repr(sorted({f["rule_id"] for f in findings})))