Escape sequences like DA responses can end up fragmented across multiple TCP/IP packets

[primeos-work]

Windows Terminal version

1.22.12111.0

Windows build number

10.0.26100.0

Other Software

OpenSSH_for_Windows_9.5p1 (less likely with OpenSSH_for_Windows_9.8p2)
tmux 3.5a

Steps to reproduce

By repeatedly (re)attaching to a tmux session after connecting to a Linux system via SSH. The chances of the bug occurring depend on multiple factors like the RTT and the tmux escape-time. The bug is easier to trigger with an OpenSSH for Windows version that isn't newer than 9.5p1 (still the most recent stable version; this is important since PowerShell/openssh-portable#771 improves things on the SSH side and is released in v9.8.2.0+). It also helps if the RTT to the SSH host is higher and if the tmux escape-time is very low (I use 0/1 but 10 should be fine as well and I even get it rarely with 50 ms). The tmux version should also be at most 3.5a (currently still the most recent version) since tmux/tmux@817b621 should be a reliable workaround for this Windows Terminal issue on the tmux side.

Expected Behavior

Everything works normally like with many other Terminal emulators. No "strange characters" (parts of the device-attributes reply escape sequence) end up in the shell prompt (I don't remember ever hitting this issue from my Linux systems).

Actual Behavior

When attaching to a tmux session, it's possible that part of the device-attributes (DA) escape sequence ends up in the active shell prompt instead of being received/interpreted by tmux.

An example:

mweiss@vm-debian-1 ~ $ 0;10;1c

Another example:

[michael@groot ~]$ 61;6;7;21;22;23;24;28;32;42c

Additional information and examples can be found here: PowerShell/Win32-OpenSSH#2275

The issue seems to be that the escape sequence can end up being split across multiple (two) TCP/IP packets and doesn't get interpreted correctly by tmux if the next TCP/IP packet exceeds the escape-time setting. I couldn't check this in the Windows Terminal source-code but the issue is likely that Windows Terminal doesn't ensure that the whole escape sequence is sent "synchronously" via a single write syscall over the SSH/TCP/IP connection. Could someone confirm this and is there a way to implement this (my Linux terminal emulators (e.g. Alacritty) do this so I never hit this bug there while it regularly (daily) occurs when using Windows Terminal). I also wasn't able to trigger this bug with Alacritty on Windows so it should be possible to ensure that the whole DA reply escape sequence ends up in a single network packet. The terminal emulator Foot seems to use the following code to ensure this: https://codeberg.org/dnkl/foot/src/commit/ed7652db5056c3658afbc11c04e164e77da18650/terminal.c#L115-L138

[similar-issues-ai]

We've found some similar issues:

• Escape sequence leaking to terminal on starting/attaching tmux (over ssh) #17439 , similarity score: 84%

If any of the above are duplicates, please consider closing this issue out and adding additional context in the original issue.

Note: You can give me feedback by 👍 or 👎 this comment.

[primeos-work]

I originally reported this issue as PowerShell/Win32-OpenSSH#2275 but will close that issue in favor of this one as we noticed that the source of this issue should be found in the Windows Terminal source code instead (that analysis/discussion starts at PowerShell/Win32-OpenSSH#2275 (comment)) and with PowerShell/openssh-portable#771 this can probably be considered fixed on the Win32-OpenSSH side.

There are quite a few duplicate issues here (and in other repos, forums, etc.) but they've been closed without correctly identifying the source of this issue:

• ConPTY(?) inserts broken control codes into tmux sessions. #7185
• Terminal putting ANSI Escape Code in terminal line when attaching tmux session #16384
• Escape sequence leaking to terminal on starting/attaching tmux (over ssh) #17439
• Printing specific sequence causes device attributes sequence to appear #19120 (might also be a result of the timing/fragmentation)

[lhecker]

Referencing the analysis you linked:

One thing Windows Terminal and ConPTY folks might want to check is if their device-attributes response is really being sent out via a single write() system call (as ESC sequences always should be).

This doesn't make sense to me. First, we have been doing that for a long time.
More importantly though, the PTY that underlies a terminal is essentially a binary ring buffer which has no idea about the boundaries of escape sequences in it. This is true on both Windows and Linux (and certainly on any other NIX). This means that all sorts of reasons can cause escape sequences to be chunked up, including into broken UTF8 sequences. Applications should handle it by implementing a proper finite state machine for parsing them. Many applications don't do this properly, including application you may assume to be of high quality. That's their fault though, because this aspect of PTYs is very old.

If not, splitting such a response [...] that do not arrive within the 10 ms patience time that tmux-3.5 allows before it concludes that this might have been a manual press of the ESC key and not a response sequence.

To my knowledge this issue also occurs with Linux terminals when using tmux via high latency SSH. As mentioned above, we do write the DA1 response atomically, so is there really anything else we can do here? I don't think so.

It is one wide string L"?61;4;6;7;14;21;22;23;24;28;32;42c", except for the initial ESC [, which all may have to go through a UTF-8 encoder, and thus might end up in separate write() system calls and separate TCP packets?

That's not how our system works. Responses get accumulated into the ControlCore::_pendingResponses buffer and then flushed out all at once here:

terminal/src/cascadia/TerminalControl/ControlCore.cpp

Lines 2174 to 2178 in 52262b0

	if (!_pendingResponses.empty())
	{
	_sendInputToConnection(_pendingResponses);
	_pendingResponses.clear();
	}

---

To my understanding there are two issues involved here:

• OpenSSH used to do a bad job at reading stdin, which was "fixed" by Increase stdin chunksize from 16 to 1024 chars PowerShell/openssh-portable#771
  (Buffer size should be even higher, but the architecture is not amendable to such changes)
• ConPTY (= our PTY) had a race condition: Fix a major stdin wakeup race condition #18816

So, try using the latest Windows Terminal releases from yesterday and see if the issue still occurs with output such as 0;10;1c.

[primeos-work]

Huge thanks @lhecker for your reply/analysis! ❤️

Many applications don't do this properly, including application you may assume to be of high quality.

Yeah, I'm unfortunately not familiar with the standards but I'd also consider it a tmux issue as well. It should luckily be fixed with tmux/tmux@817b621 now but it'll take a long time until that's available on most SSH hosts.

To my knowledge this issue also occurs with Linux terminals when using tmux via high latency SSH.

Ok, so you've also experienced that issue on Linux systems?
I don't remember ever running into that issue from my Linux laptops but it's certainly possible - I guess it depends on the terminal emulator (even though it's not necessarily a terminal emulator bug but it looks like terminal emulators can at least do a better or worse job at preventing it by ensuring the response ends up in a single network packet or at least using large enough buffer sizes - I'm not familiar at all with the involved details though).

As mentioned above, we do write the DA1 response atomically, so is there really anything else we can do here? I don't think so.

Ok, I was hoping there would be a way to significantly improve the chances of the response ending up in a single network packet but we can close this issue then if there's no reasonable improvement left.

That's not how our system works. Responses get accumulated into the ControlCore::_pendingResponses buffer and then flushed out all at once here:

Ok, thanks. If I followed the calls correctly it'll end up here?:

terminal/src/terminal/adapter/InteractDispatch.cpp

Lines 30 to 42 in 91c9a14

	// Method Description:
	// - Writes a collection of input to the host. The new input is appended to the
	// end of the input buffer.
	// If Ctrl+C is written with this function, it will not trigger a Ctrl-C
	// interrupt in the client, but instead write a Ctrl+C to the input buffer
	// to be read by the client.
	// Arguments:
	// - inputEvents: a collection of IInputEvents
	void InteractDispatch::WriteInput(const std::span<const INPUT_RECORD>& inputEvents)
	{
	const auto& gci = ServiceLocator::LocateGlobals().getConsoleInformation();
	gci.GetActiveInputBuffer()->Write(inputEvents);
	}

I guess one last thing that could theoretically be done to avoid this issue / improve the situation could be to let the active input buffer fully drain before appending such "atomic" escape sequences (or some alternative means to ensure the whole response ends up in a single 1024 bytes buffer) but I have no idea how well that would fit into the code and it would obviously have the drawback of slowing things down by not utilizing the full buffer size in those cases and (significantly) increasing complexity so it's most likely not worth it...

I guess in the end it really is a tmux issue/problem (I also couldn't reproduce it with GNU Screen when I tried it just now but I didn't check if they also issue a DA request but I guess it's very likely the case) as Windows Terminal probably doesn't violate any standards here and I just incorrectly blamed it on Windows Terminal / Win32-OpenSSH as I didn't hit it with other setups (Linux systems/terminals). Sorry about that! (There are already quite a few reports though as mainly Windows Terminal users seem to run into this issue. I guess we at least have it documented/analyzed quite extensively now...)

(Buffer size should be even higher, but the architecture is not amendable to such changes)

Ok, that's a bit of a bummer but fair. With a higher buffer size the issue might already be gone almost entirely.

So, try using the latest Windows Terminal releases from yesterday and see if the issue still occurs with output such as 0;10;1c.

I'll give it a go but unfortunately I doubt it'll improve things as apparently your fix already got backported to version 1.22 and ea0ba56 is in version 1.22.12111.0 that I used for the fresh tests.

[lhecker]

Ok, so you've also experienced that issue on Linux systems?

Not really. I've generally only administered servers that are either in a local DC or in my home. But I know it's an issue, which is why the escape-time setting exists at all.

Ok, thanks. If I followed the calls correctly it'll end up here?:

Sort of. That's the PTY side of things where input is written into the aforementioned ring buffer. The architecture is bad, so don't take the following as "it should be like this" but more like "it's currently like this":

• Your app write DA1 into stdout (a pipe)
• The PTY (aka ConPTY aka OpenConsole.exe aka conhost.exe) reads from stdout and
  • before v1.22:
    • Parses stdout into VT sequences
    • Reassembles them back into VT output
    • Sends it to the terminal (another pipe)
  • since v1.22:
    • Sends it to the terminal (another pipe)
• The terminal reads from the PTY and
  • Parses the VT sequences
  • Sees the DA1
  • Writes the response into _pendingResponses
• Now it writes _pendingResponses back to the PTY (a pipe)
• The PTY reads from stdin (terminal) and
  • Parses stdin into VT sequences (like above before v1.22 == bad, needs a refactor)
  • Writes the input events into a "ring buffer" / queue
  • Signals availability of input to your app's stdin pipe
    • This is where Fix a major stdin wakeup race condition #18816 happened
• Your app gets woken up because stdin is available and
  • Reads stdin from the PTY ring buffer

I guess one last thing that could theoretically be done to avoid this issue / improve the situation could be to let the active input buffer fully drain before appending such "atomic" escape sequences

I'd be against that. Applications should implement better VT parsers (which are robust against this), and terminals should implement the kitty keyboard protocol (including us; it turns Esc keys into the unmistakable sequence \x1b[27u). This fixes the issue from both sides IMO. Particularly the latter is a good fix, because your suggestion and any other current workaround rely on heuristics which is not great.

There are already quite a few reports though as mainly Windows Terminal users seem to run into this issue.

Windows Terminal has had (and still has) various major bugs with its VT handling, so the attribution is totally fair IMO. Most notably, before #17510 (= before v1.22) we would mangle VT sequences in stdout. We were the only PTY and terminal to do this, and it resulted in lots of issues. We also still somewhat mangle stdin but that's unrelated to this issue. I'm also intending to fix that in the near term.

I'll give it a go but unfortunately I doubt it'll improve things as apparently your fix already got backported to version 1.22 and ea0ba56 is in version 1.22.12111.0 that I used for the fresh tests.

Hmm that's weird then... Your second example looks like just the Esc character was mistaken for an Esc key so it printed the whole DA1 response to your prompt. But the first example looked exactly like how #18816 presented itself. We may have to investigate this, although I'm not sure how. (It doesn't happen to me so I can't debug it. I also could only figure out #18816 after I was able to reproduce it in wezterm.)