Skip to content

Commit beeac20

Browse files
wu-shengclaude
andcommitted
fix: never create a gRPC channel in the Gunicorn prefork master
With grpcio >= 1.80 (EventEngine) a gRPC channel that lives across fork() breaks: continuous `Kick Failure (eventfd_write: Bad file descriptor)` stderr spam and a racy silent deadlock of forked workers inside gRPC's own at-fork handlers; grpcio 1.83.0 removed the last legacy-poller opt-out (grpc/grpc#42828; upstream reports grpc/grpc#43055 and grpc/grpc#43062, both open and unfixed). - `sw-python run -p gunicorn`: the master now only installs instrumentation (start_prefork_master) and arms the fork hooks; queues, reporter threads and the gRPC channel are created in each forked worker only (the model uWSGI has always used). The master no longer registers as a service instance. - New agent.started() guard: instrumented code no-ops (NoopSpan, dropped logs/meters) in a process whose reporters are not active, protecting `gunicorn --preload` app imports in the master. - GRPC_POLL_STRATEGY is no longer set; the grpcio floor is raised to >= 1.83 (generated stubs already require it at import) and codegen grpcio-tools is pinned in lockstep so future wheels cannot silently raise the runtime floor. - Gunicorn prefork + SW_AGENT_ASYNCIO_ENHANCEMENT is rejected instead of starting an unsafe pre-fork agent. - New plugin tests: sw_gunicorn (full consumer -> gunicorn-provider trace with CrossProcess ref, exact worker-boot-count and no-Kick-Failure log assertions, async-rejection case) and sw_fork_support (explicit os.fork() with a continuous cross-fork trace via SW_AGENT_EXPERIMENTAL_FORK_SUPPORT). Validated in Docker on grpcio 1.80.0 / 1.82.1 / 1.83.0 x 8-10 boots each: zero error spam, zero worker deadlocks, all segments delivered; fd-level inspection confirms the master holds no collector connection while each worker holds exactly one. Fixes apache/skywalking#13958 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent 88d30ab commit beeac20

24 files changed

Lines changed: 830 additions & 180 deletions

Makefile

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -50,14 +50,16 @@ else
5050
-curl -sSL https://install.python-poetry.org | python3 -
5151
endif
5252

53+
# grpcio-tools pinned: generated stubs refuse grpcio older than the tools version,
54+
# keep in sync with the grpcio floor declared in pyproject.toml
5355
.PHONY: gen
5456
gen:
55-
poetry run pip install 'grpcio-tools>=1.68.0' packaging
57+
poetry run pip install 'grpcio-tools~=1.83.0' packaging
5658
poetry run python3 tools/grpc_code_gen.py
5759

5860
.PHONY: gen-basic
5961
gen-basic:
60-
python3 -m pip install 'grpcio-tools>=1.68.0' packaging
62+
python3 -m pip install 'grpcio-tools~=1.83.0' packaging
6163
python3 tools/grpc_code_gen.py
6264

6365
.PHONY: install

docs/en/setup/CLI.md

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -53,7 +53,7 @@ or
5353

5454
`uwsgi --die-on-term --http 0.0.0.0:5000 --http-manage-expect --master --workers 3 --enable-threads --threads 3 --manage-script-name --mount /=main:app`
5555

56-
Please change it to (**the `-p` option starts one agent in each process, which is the correct behavior**):
56+
Please change it to (**the `-p` option starts one agent in each worker process and none in the master, which is the correct behavior**):
5757

5858
**Important:** if the call to uwsgi/gunicorn is prefixed with other commands, this approach will fail
5959
since agent currently looks for the command line input at index 0 for safety as an experimental feature.
@@ -71,6 +71,9 @@ Note that `sw-python` also work with spawned subprocess (os.exec*/subprocess) as
7171

7272
Additionally, `sw-python` started agent works well with `os.fork` when your application forks workers,
7373
as long as the `SW_AGENT_EXPERIMENTAL_FORK_SUPPORT` is turned on. (It will be automatically turned on when gunicorn is detected)
74+
Avoid calling `os.fork()` while the agent is actively talking to the collector (e.g. immediately at startup, during
75+
registration): a fork during an in-flight gRPC call can trip gRPC's own at-fork handling (see grpc/grpc#43055) and
76+
hang the child. Forking a moment after startup, or between requests, is safe.
7477

7578
## Configuring the agent
7679

docs/en/setup/Configuration.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ export SW_AGENT_YourConfiguration=YourValue
3838
| agent_collector_heartbeat_period | SW_AGENT_COLLECTOR_HEARTBEAT_PERIOD | <class 'int'> | 30 | The agent will exchange heartbeat message with SkyWalking OAP backend every `period` seconds |
3939
| agent_collector_properties_report_period_factor | SW_AGENT_COLLECTOR_PROPERTIES_REPORT_PERIOD_FACTOR | <class 'int'> | 10 | The agent will report service instance properties every `factor * heartbeat period` seconds default: 10*30 = 300 seconds |
4040
| agent_instance_properties_json | SW_AGENT_INSTANCE_PROPERTIES_JSON | <class 'str'> | | A custom JSON string to be reported as service instance properties, e.g. `{"key": "value"}` |
41-
| agent_experimental_fork_support | SW_AGENT_EXPERIMENTAL_FORK_SUPPORT | <class 'bool'> | False | The agent will restart itself in any os.fork()-ed child process. Important Note: it's not suitable for short-lived processes as each one will create a new instance in SkyWalking dashboard in format of `service_instance-child(pid)`. This feature may not work when a precise combination of gRPC + Python 3.7 + subprocess (not fork) is used together. The agent will output a warning log when using on Python 3.7 for such a reason. |
41+
| agent_experimental_fork_support | SW_AGENT_EXPERIMENTAL_FORK_SUPPORT | <class 'bool'> | False | The agent will restart itself in any os.fork()-ed child process. Important Note: it's not suitable for short-lived processes as each one will create a new instance in SkyWalking dashboard in format of `service_instance-child(pid)`. When the sw-python CLI detects a pre-forking server (Gunicorn), only worker processes run a full agent; the master installs instrumentation only. |
4242
| agent_queue_timeout | SW_AGENT_QUEUE_TIMEOUT | <class 'int'> | 1 | DANGEROUS - This option controls the interval of each bulk report from telemetry data queues Do not modify unless you have evaluated its impact given your service load. |
4343
| agent_asyncio_enhancement | SW_AGENT_ASYNCIO_ENHANCEMENT | <class 'bool'> | False | Replace the threads to asyncio coroutines to report telemetry data to the OAP. This option is experimental and may not work as expected. |
4444
### SW_PYTHON Auto Instrumentation CLI

docs/en/setup/faq/How-to-use-with-gunicorn.md

Lines changed: 17 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ serves requests.
1414

1515
> Note: Python 3.10+ is required. Earlier versions (3.7-3.9) are no longer supported.
1616
17-
**TL;DR:** specify `-p` or `--prefork` in `sw-python run -p` and all Gunicorn workers and master will get their own working agent.
17+
**TL;DR:** specify `-p` or `--prefork` in `sw-python run -p` and every Gunicorn worker will get its own working agent (the master is instrumented only and runs no agent).
1818

1919
**Important:** if the call to gunicorn is prefixed with other commands, this approach will fail
2020
since agent currently looks for the command line input at index 0 for safety as an experimental feature.
@@ -29,17 +29,28 @@ sw-python run -p gunicorn gunicorn_consumer_prefork:app --workers 2 --worker-cla
2929
By specifying the -p or --prefork option in sw-python CLI, the `agent_experimental_fork_support` agent option will be turned on automatically.
3030

3131
Startup flow:
32-
sw-python -> gunicorn -> master process (agent starts) -> fork -> worker process (agent restarts due to os.register_at_fork)
32+
sw-python -> gunicorn -> master process (instrumentation only) -> fork -> worker process (full agent starts due to os.register_at_fork)
3333

34-
The master process will get its own agent, although it won't report any trace, since obviously it doesn't take requests,
35-
it still reports metrics that is useful for debugging
34+
The master process does not run a full agent: it only installs instrumentation, and the reporters plus the
35+
gRPC channel are created in each forked worker. Therefore the master does not appear as a service instance
36+
(it takes no requests anyway). A gRPC channel created before fork() is unsafe with grpcio >= 1.80, see
37+
[apache/skywalking#13958](https://github.com/apache/skywalking/issues/13958).
3638

3739
> A runnable example can be found in the demo folder of skywalking-python GitHub repository
3840
41+
### Known issue with agent <= 1.2.0 and grpcio >= 1.80
42+
43+
Agent versions up to 1.2.0 started a full agent (including a gRPC channel) in the Gunicorn master before forking.
44+
With grpcio >= 1.80 this produces continuous `Kick Failure (eventfd_write: Bad file descriptor)` errors
45+
and can silently hang workers. Tracing generally keeps working — the errors come from the gRPC client polling
46+
engine and are unrelated to the OAP version. Workarounds on old agents: pin `grpcio<1.80` or use `SW_AGENT_PROTOCOL=http`.
47+
Fixed agent versions require `grpcio >= 1.83` and never create a gRPC channel in the master.
48+
3949
## Manual Approach (only use when sw-python doesn't work)
4050

41-
**Limitation**: Using normal postfork hook will not add observability to the master process, you could also define a prefork hook to
42-
start an agent in the master process, with a instance name like `instance-name-master(<pid>)`
51+
**Limitation**: Using normal postfork hook will not add observability to the master process.
52+
Do NOT start an agent in the master process (e.g. from a prefork hook): a gRPC channel created before
53+
fork() is unsafe with grpcio >= 1.80 and can deadlock workers.
4354

4455
The following is just an example, since Gunicorn's automatic injection approach is likely to work in many situations.
4556

docs/en/setup/faq/How-to-use-with-uwsgi.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,8 +13,8 @@ Some of the original discussion can be found here:
1313
> You can always fall back to the manual approach.
1414
> (although it's also possible to pass postfork hook without changing code, which is essentially how sw-python is implemented)
1515
16-
> Limitation: regardless of the approach used, uWSGI master process cannot be safely monitored. Since it doesn't take any requests, it is generally acceptable.
17-
> Alternatively, you could switch to Gunicorn, where its master process can be monitored properly along with all child workers.
16+
> Limitation: regardless of the approach used, the uWSGI master process is not monitored. Since it doesn't take any requests, it is generally acceptable.
17+
> Gunicorn behaves the same way — only worker processes are monitored, which is the fork-safe design.
1818
1919
**Important**: The `--enable-threads` and `--master` option must be given to allow the usage of post_fork hooks and threading in workers.
2020
In the `sw-python` CLI, these two options will be automatically injected for you in addition to the post_fork hook.

0 commit comments

Comments
 (0)