Multipath on QWFWD.

[tibazera]

Multipath UDP for QWFWD

PR: Add multipath upstream routing with automatic failover (ExitLag/AntiLag-style)

Problem

When the single upstream path between qwfwd and the QW server becomes
congested or fails, every client behind the proxy is disconnected.
There is no way to survive transient route failures without dropping sessions.

Solution

This PR adds Multipath UDP support (src/multipath.c / src/multipath.h).

Up to 4 independent UDP sockets are opened to the server (or to relay
endpoints). Each path is continuously probed for RTT and packet loss.
Game traffic flows on the single best-scored path at any given moment.
If that path degrades or dies, traffic seamlessly switches to the next
best path — the client never sees a disconnect.

                    ┌─────────────────────────────────────────┐
  Client (ezQuake)  │             qwfwd proxy                 │  QW Server
                    │                                         │
  ─── UDP ─────────▶│  peer_t                                 │
                    │   └─ mp_ctx_t                           │
                    │        ├─ path0 [ACTIVE ★]  ──UDP──────▶│ port 27500
                    │        ├─ path1 [ACTIVE  ]  ──UDP──────▶│ port 27500
                    │        ├─ path2 [PROBING ]  ──UDP──────▶│ port 27500
                    │        └─ path3 [DEAD    ]              │
                    └─────────────────────────────────────────┘

How it works

Path states

DISABLED ──configure──▶ PROBING ──first RTT──▶ ACTIVE
                                                  │
                              ┌───────────────────┤
                              │ loss > 30%         │ no response > 3s
                              ▼                    ▼
                          DEGRADED             DEAD ──10s──▶ REVIVING
                              │                                   │
                              │ loss < 10%         first RTT ─────┘
                              └──────▶ ACTIVE

RTT measurement

A tiny 8-byte probe packet is sent on every path every 500 ms
(configurable via mp_probe_interval).
The server echoes it back unchanged.
RTT is computed using RFC 6298 EWMA:

SRTT   = (1 - 0.125) * SRTT   + 0.125 * sample
RTTVAR = (1 - 0.25)  * RTTVAR + 0.25  * |SRTT - sample|

Path selection (score)

score = SRTT_ms + 200 * loss_rate

Bonus/penalty	Value
DEGRADED path	+150 ms
PROBING path (unknown)	fixed 500 ms
DEAD / DISABLED	∞ (excluded)

The active path switches when:

• The current path dies, or
• An alternative path score < current score × 0.75 (25% improvement)

Deduplication

Each path has its own socket fd. When the active path switches, a brief
window exists where packets from the old path may still arrive. A
64-slot circular cache of server→client sequence numbers discards
duplicates before they reach the client.

---

Configuration

qwfwd.cfg / server cvars

# Disable (default – identical to current behaviour)
cl_multipath 0

# Auto mode: open MP_MAX_PATHS direct sockets to the server.
# Useful when the proxy host has multiple NICs or routing policies.
cl_multipath 1

# Explicit relay list (comma-separated  ip:port)
# Each entry becomes one upstream path.
cl_multipath "10.0.0.1:27500,10.0.0.2:27500,10.0.1.1:27500"

# Tuning (optional)
mp_probe_interval   500     # ms between RTT probes
mp_switch_threshold 0.75    # switch factor (lower = more aggressive switching)
mp_dead_timeout     3000    # ms without response -> DEAD
mp_revive_interval  10000   # ms before retrying a DEAD path

ezQuake client side

No changes required on the client. The client still connects normally
via cl_proxyaddr. The multipath logic is entirely server-side inside
qwfwd.

---

Console commands

mp_status          – print per-peer path state, RTT, loss, packet counters
mp_reset <ip>      – force-reinitialise multipath for a specific peer

Example output:

Peer 203.0.113.42:
Multipath: 3 path(s), switches=2
  [0] path0    ACTIVE   srtt= 24.3ms rttvar=  1.2ms loss=  0% tx=1420 rx=1418 <-- ACTIVE
  [1] path1    ACTIVE   srtt= 31.7ms rttvar=  3.4ms loss=  2% tx=1420 rx=1391
  [2] path2    DEAD     srtt=  0.0ms rttvar=  0.0ms loss=  0% tx=  24 rx=   0

---

Files changed

File	Change
src/multipath.h	New – data structures, public API
src/multipath.c	New – full implementation
src/peer.h	Add mp_ctx_t mp and mp_configured to peer_t
src/peer.c	Call MP_Init() on new peer, MP_Shutdown() on remove, MP_Frame() in peer loop + secondary socket drain
src/clc.c	Configure multipath on S2C_CONNECTION, switch NET_SendPacket to NET_SendPacketMP
src/net.c	Add NET_SendPacketMP() wrapper
src/cmd.c	Register mp_status and mp_reset commands
src/cvar.c	Register cl_multipath and tuning cvars
CMakeLists.txt	Add multipath.c to sources, link -lm

---

Testing

# Build
cmake -B build && cmake --build build

# Run with 2 explicit paths (both pointing at same server = duplicate
# routing; replace IPs with actual relay/tunnel endpoints in production)
./build/qwfwd +set cl_multipath "qw.server.com:27500,relay.example.com:27500"

# In another terminal, watch path state
./build/qwfwd_console mp_status

Simulating path failure

# Drop traffic on a specific path using iptables (Linux)
iptables -A OUTPUT -d <path1_ip> -p udp --dport 27500 -j DROP

# qwfwd should detect DEAD within mp_dead_timeout ms and switch
# Remove the rule to trigger REVIVING -> ACTIVE
iptables -D OUTPUT -d <path1_ip> -p udp --dport 27500 -j DROP

---

Backward compatibility

• When cl_multipath 0 (default), the code path is identical to the
  pre-PR behaviour. Zero overhead.
• No protocol changes visible to the QW server or to clients.
• Probe packets are directed at the server's UDP port; any server that
  does not recognise the probe magic simply discards them (8 bytes,
  no side effects).

---

Future work

• DNS resolution in MP_ParseConfig (currently requires dotted-decimal IPs)
• Per-path bandwidth throttling
• QUIC-style connection migration (send migration hint to server on path switch)
• Expose path stats via QTV status API

multipath.c
qwfwd-multipath.patch

qwfwd-multipath.patch

[qqshka]

Why not just a PR request?

I tried to understand how this works a little, but I gave up.

qwfwd communicates with the server (or with another qwfwd) after establishing a session, it's a stateful protocol, you can't just send a packet to an arbitrary qwfwd without establishing a session, I don't understand how what you're showing could work.

Also, I don't understand what cl_multipath "relay1:port,relay2:port,..." is, how are relay1, relay2, etc. determined?

At best, this could work if the machine really has multiple WAN interfaces, then yes, you could send packets from different interfaces, but even then, I don't remember exactly, but most likely the peer just wouldn't understand that.

[mihax86]

I think I understand, he means to have a backup route for when the best route goes offline or becomes laggy, the thing is, the route must change without the user even noticing. Too rare of a edge case. I don't think its worth the effort, but it's a great challenge for those interested.

[tibazera]

Sorry qqshka, im using a paid IA to try do a better code to going similar to Exitlag, thats why i posted here.
Next what your answer:

scriptsmultipath.zip

Multipath UDP for ezQuake + qwfwd

Background: why the first proposal was wrong

The first proposal tried to add multipath upstream — between qwfwd and the
QW server. The qwfwd developer correctly rejected it.

The QuakeWorld server identifies a client session by (remote IP, qport).
If qwfwd sent packets from different source IPs, the server would either drop
them or treat them as new, unrelated clients. There is no way around this
without modifying every QW server in existence.

The correct layer is between ezQuake and qwfwd, where we control both
ends. The QW server never sees anything change.

---

Two topologies, one protocol

This design supports two deployment topologies. Both use the exact same
wire protocol and the same code. The difference is only in how qwfwd
instances are arranged.

---

Topology 1 — Multiple paths, single qwfwd

The simplest case. The player's machine has more than one internet connection
(two ISPs, ISP + 4G, ISP + VPN tunnel, etc.), or the proxy host has multiple
WAN interfaces.

ezQuake
  ├─ socket0 ──── ISP route  ─────╮
  ├─ socket1 ──── 4G route   ─────▶  qwfwd  ────▶  QW Server
  └─ socket2 ──── VPN route  ─────╯
       (different source IPs,        (single session,
        same destination)             single upstream IP)

All three packets arrive at the same qwfwd from different source IPs.
qwfwd uses the session token to recognise they belong to the same client,
and forwards them over its single upstream QW session.

Limitation: All paths converge at the same physical machine. If that
machine or its upstream link fails, the client disconnects regardless of
how many paths exist. Useful for reducing latency spikes, not for
surviving a proxy outage.

---

Topology 2 — Geographically distributed proxies (relay mode)

The more powerful case. Multiple qwfwd instances in different cities act as
entry points. A central qwfwd (the relay) holds the single upstream session
with the QW server.

ezQuake
  ├─ socket0 ──▶ qwfwd-BR (São Paulo)  ──╮
  ├─ socket1 ──▶ qwfwd-US (Miami)      ──▶  qwfwd-RELAY (Amsterdam) ──▶ QW Server
  └─ socket2 ──▶ qwfwd-EU (Frankfurt)  ──╯

  edge proxies: stateless,          relay: holds the single
  geographically close to player    QW session with the server

Why this matters: The internet between São Paulo and Amsterdam has
multiple routes. At different times of day, the US path might be faster,
or the direct BR→EU path. The player's client measures all three in real
time and picks the best one. If any edge proxy or its upstream link fails,
traffic automatically moves to another path within milliseconds. The player
never disconnects.

This is exactly what commercial services like ExitLag do, but built into
the open-source QW toolchain.

---

Why existing qwfwd chaining does not already solve this

qwfwd already supports chaining: cl_proxyaddr "proxyA@proxyB" makes
ezQuake connect through proxyA which connects through proxyB to the server.

The key difference:

Feature	Today's chaining	This proposal
Number of active paths	1	Up to 4
Path selection	Static (set at connect time)	Dynamic (measured per frame)
Failover	None (chain breaks = disconnect)	Automatic (< 1 second)
Edge proxy state	Full session required	Stateless forwarder

With today's chaining, proxyA and proxyB each establish their own session.
If you wanted three parallel chains, you'd have three clients on the server.
The relay mode solves this by having only one qwfwd (the relay) talk to the
server. Edge proxies just forward bytes with the MP header intact — they are
stateless and require no per-session configuration.

---

Wire protocol (identical for both topologies)

Handshake

ezQuake adds two keys to its userinfo during the QW connect packet:

"*mp_token"  "a3f8b201c9d47e56"   // 8 random bytes as hex, unique per session
"*mp_paths"  "3"                  // number of paths the client will open

The first qwfwd that receives the connect (edge proxy or direct) reads
these keys, stores the token, and echoes *mp_token back in the connect
response. ezQuake checks for the echo — if absent, it falls back silently
to single-path mode.

This means old qwfwd instances work without any changes. The client
just won't get multipath through them, and nothing breaks.

Per-packet header (6 bytes)

After the session is established, ezQuake prepends a 6-byte header to every
UDP datagram it sends to the proxy:

 0       1       2       3       4       5
 +-------+-------+-------+-------+-------+-------+
 | magic = 0x5157 ('QW') |  ver  | path  |  rsvd |
 +-------+-------+-------+-------+-------+-------+
   2 bytes (big-endian)    1 byte  1 byte  2 bytes

• magic 0x5157: identifies this as a multipath packet. A qwfwd that
  does not understand this will discard it; ezQuake detects this via the
  missing echo and falls back to single-path.
• version 0x01: allows future protocol changes without breaking old
  implementations.
• path_id 0..3: which socket sent this packet. The relay uses this to
  know which edge proxy address to reply to, and ezQuake uses it to measure
  per-path RTT.
• reserved: must be zero, reserved for future use.

The header is stripped by the relay (or single qwfwd in Topology 1) before
forwarding to the QW server. The server sees a completely standard QW
packet.

Return path: the relay prepends the same header on every reply to the
client, setting path_id to whichever path the most recent client packet
arrived on. This lets ezQuake measure RTT per path using the sequence
numbers already in the QW NetChannel header — no extra probe packets are
needed during active gameplay.

---

qwfwd operating modes

Mode A — direct (existing behaviour, unchanged)

Receives from client, forwards to server. If an MP header arrives,
the magic is unknown to old qwfwd and the packet is discarded. The client
falls back to single-path. Zero risk of breaking existing deployments.

Mode B — mp-endpoint (Topology 1)

Receives MP-wrapped packets directly from ezQuake. Uses the session token
to identify the peer regardless of source IP. Strips the header, forwards
to the QW server. Wraps replies with the MP header before sending back.

Enabled automatically when a client connects with *mp_token in userinfo.
No configuration needed.

Mode C — mp-edge (Topology 2, edge proxies)

Receives MP-wrapped packets from ezQuake and forwards them with the header
intact to the relay. Does not establish a session with the QW server.
Completely stateless with respect to the MP session.

# qwfwd-BR, qwfwd-US, qwfwd-EU:
mp_relay "relay.example.com:27500"

When mp_relay is set, qwfwd automatically operates as an edge proxy for
any packet carrying an MP header.

Mode D — mp-relay (Topology 2, relay)

Receives MP-wrapped packets from edge proxies. Uses the token embedded in
the header to find the upstream peer session. Strips the header and forwards
to the QW server as in Mode B.

Crucially, the relay learns which edge proxy is handling each path
dynamically, from the source IP of arriving packets: "path 0 packets are
arriving from qwfwd-BR's IP, path 1 from qwfwd-US's IP." When it replies,
it sends back to the appropriate edge proxy IP. No static configuration of
edge proxy addresses is needed on the relay.

The relay does need to know the session token, which it receives during the
initial connect handshake (forwarded by the first edge proxy). Paths 1
and 2 carry only data packets — no separate handshake — and the relay
accepts them by matching the token in the MP header.

---

Full packet flow (Topology 2)

1. ezQuake generates token = "a3f8b201c9d47e56", opens 3 sockets.

2. socket0 sends QW `connect` with userinfo containing *mp_token.
   → qwfwd-BR receives, sees mp_relay is set, forwards to qwfwd-RELAY.
   → qwfwd-RELAY stores token, establishes QW session with QW Server.
   → qwfwd-RELAY echoes *mp_token in connect response.
   → ezQuake confirms multipath is active.

3. socket1 and socket2 start sending data packets with MP headers
   (path_id=1 and path_id=2). No separate handshake needed.
   → qwfwd-US and qwfwd-EU forward packets with header intact.
   → qwfwd-RELAY receives packets from new source IPs (US/EU proxies),
     finds the session by token, updates path_src[1] and path_src[2].

4. QW Server replies go to qwfwd-RELAY.
   → Relay wraps reply with MP header (path_id = whichever path had
     the best recent RTT, or simply the last one that sent).
   → Sends to path_src[best_path] = qwfwd-US.
   → qwfwd-US forwards to ezQuake socket1.

5. If qwfwd-US goes down:
   → ezQuake socket1 stops receiving, RTT probe times out.
   → After MP_DEAD_TIMEOUT (3s), path1 = DEAD.
   → ezQuake switches to path0 or path2 automatically.
   → qwfwd-RELAY starts receiving only from BR and EU proxies.
   → Client never disconnects. Server never noticed.

---

Failure scenarios

Edge proxy dies

ezQuake detects the path timeout within 3 seconds and switches to the next
best path. The QW session at the relay is unaffected. Client stays connected.

Relay dies

The upstream QW session is lost. The client disconnects. This is the same
failure mode as today's single qwfwd — multipath does not eliminate it.

Mitigation: run the relay on the same machine as the QW server, or on a
reliable VPS. Edge proxies are the ones exposed to variable internet
conditions; the relay can be as stable as the server itself.

All edge proxies die simultaneously

Client disconnects, same as relay dying. In practice, this would require
a coordinated failure across multiple data centres — unlikely in normal
operation.

---

Backward compatibility

Scenario	Result
cl_multipath 0 (default)	Identical to today, zero overhead
New ezQuake + old qwfwd	No echo, client falls back to single path silently
Old ezQuake + new qwfwd	No *mp_token in userinfo, qwfwd behaves normally
Topology 1 (single qwfwd)	Works with Mode B only, no relay needed
Topology 2 (distributed)	Requires Mode C on edges + Mode D on relay

---

Summary of code changes

New files

File	Repo	Est. lines	Purpose
cl_multipath.c	ezquake	~280	All client-side multipath logic
cl_multipath.h	ezquake	~80	Structs, API, cvars
mp_session.c	qwfwd	~200	Modes B/C/D: session tracking, header handling, relay logic
mp_session.h	qwfwd	~70	Structs, API

Changes to existing files

File	Lines added	What
ezquake net_udp.c	~15	NET_MP_InjectPacket() to feed decapsulated packets into normal receive pipeline
ezquake cl_main.c	~10	Hooks for init, disconnect, send routing, per-frame recv
qwfwd peer.h	~3	Add peer_mp_t mp field to peer_t
qwfwd peer.c	~5	Init/shutdown/token-lookup hooks
qwfwd clc.c	~10	Read token from userinfo, echo back, strip header on forward
qwfwd svc.c	~2	Replace NET_SendPacket with MP_WrapAndSend on server→client path
qwfwd CMakeLists	~2	Add mp_session.c to sources, link -lm

---

Open questions for the developer

1. Token in userinfo vs OOB handshake. Putting the token in userinfo is
   the simplest approach and reuses existing infrastructure. An alternative
   is a new OOB packet (\xff\xff\xff\xffmp_hello <token> <paths>) exchanged
   before connect. This avoids polluting userinfo but adds a round-trip.

2. 6-byte header on every packet vs OOB-only probes. The header gives
   accurate per-path RTT during gameplay at the cost of 6 extra bytes per
   datagram. Without it, we would need explicit probe packets and RTT
   measurement would be less accurate during high-traffic periods.

3. Relay mode as cvar or separate binary. A cvar (mp_mode auto|edge|relay)
   keeps a single binary. A separate binary is cleaner operationally but
   duplicates code. Suggestion: cvar, auto-detected based on whether
   mp_relay is set.

4. Token hijacking on the relay. Should the relay validate that a new
   path's source IP is "expected" in some way? Options: whitelist of known
   edge proxy IPs, same-ASN check, or no validation (simplest, but allows
   session stealing if the token is intercepted). For a gaming use case,
   no validation is probably acceptable.