add: use SO_REUSEPORT on platform supporting it

[mkleczek]

Fixes #4694

Stacked on top of #5077 to make sure liveness and readiness checks are reliable.

[steve-chavez]

This is failing all tests, I'd suggest to put these type of PRs as draft.

[mkleczek]

This is failing all tests, I'd suggest to put these type of PRs as draft.

Hmm... worked before latest push. Will switch to draft and fix.

[mkleczek]

This is failing all tests, I'd suggest to put these type of PRs as draft.

Hmm... worked before latest push. Will switch to draft and fix.

@steve-chavez - it looks like the issue is that on my machine PostgREST startup is fast enough so that it loads the schema cache before it accepts any requests. Here in CI the new instance fails with 503 because it didn't yet load the schema cache.

The question: is there any particular reason why we return 503, instead of simply not start listening on the socket until schema cache is loaded? The way we have it right now means we can't really support zero-downtime upgrades because once we start the new instance but before it loads the schema cache, some clients will get 503.

[steve-chavez]

The question: is there any particular reason why we return 503, instead of simply not start listening on the socket until schema cache is loaded?

We were aiming to have requests wait instead of 503, this waiting does happen during schema cache reload but not on startup; we discussed this on #4129. Would it be better to not listen on the socket? How would clients behave in this case?

[develop7]

They would get a "connection refused" error, which means nobody there and is imo more confusing that any 5xx error. UX-wise I would prefer some waiting to a presumably hard fail any day.

[mkleczek]

We were aiming to have requests wait instead of 503, this waiting does happen during schema cache reload but not on startup; we discussed this on #4129. Would it be better to not listen on the socket? How would clients behave in this case?

They would get a "connection refused" error, which means nobody there and is imo more confusing that any 5xx error. UX-wise I would prefer some waiting to a presumably hard fail any day.

I am not convinced, see below.

This is a complex topic so let's dig into it a little more. The startup sequence right now is:

1. Postgrest is not running.
2. Clients get connection refused
3. Postgrest starts listening on a socket
4. Clients get 503
5. Postgrest loaded schema cache
6. Normal traffic

The alternatives are:
a - blocking during schema cache loading

1. Postgrest is not running.
2. Clients get connection refused
3. Postgrest starts listening on a socket
4. Clients are blocked and potentially timeout getting some network error
5. Postgrest loaded schema cache
6. Normal traffic

b - listening on a socket only after schema cache loaded

1. Postgrest is not running.
2. Clients get connection refused
3. Postgrest loaded schema cache and starts listening
4. Normal traffic

So from the point of view of the clients (they don't know when Postgrest was started), we have 3 alternatives:

1. connection refused -> 503 -> normal
2. connection refused -> blocked/time out -> normal
3. connection refused -> normal

I am not sure what value clients get from the first two options comparing to the third one. Diagnostics and readiness checks should be done using admin server anyway.

In case of SO_REUSEPORT the situation is even worse if we start listening early. We have the following situation: instance 1 is running, instance 2 is started. From the point of view of clients:

1. Today: normal traffic -> some clients get 503 (both instances are listening only one is ready) -> normal traffic
2. Blocking: normal traffic -> some clients blocked/time out (both instances are listening one is blocking) -> normal traffic
3. Not listening: just normal traffic (there are no disruptions at all because all requests are handled by instance 1 until instance 2 is ready and starts listening).

So the first two options cause disruptions whereas the third one is fully zero-downtime and transparent to the clients.

My take on it would be:

1. Start admin server as early as possible.
2. Load schema cache.
3. Start listening on main socket.

This would require splitting binding from listening on the main socket (ie. we need to bind without listening first so that we can pass the socket to the admin server).

@steve-chavez @develop7 thoughts?

[mkleczek]

Dependent on resolving (or having a workaround to) yesodweb/wai#853

[steve-chavez]

So the first two options cause disruptions whereas the third one is fully zero-downtime and transparent to the clients.
My take on it would be:

Agree, sounds much better.

[steve-chavez]

Besides the last nit on docs, this looks good to go. Great work! 🚀

[steve-chavez]

Hm, for some reason I'm unable to merge. It says the branch is out-of-date with the base branch?

[mkleczek]

Hm, for some reason I'm unable to merge. It says the branch is out-of-date with the base branch?

Let me check if it is not multi-parent commit by accident.

[steve-chavez]

@mkleczek There was a hidden comment that was unresolved (had to click on X hidden items to reveal), for some reason it didn't appear on the "Files changed" tab. Sorry about that. Merging now.