You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
fix: never create a gRPC channel in the Gunicorn prefork master
With grpcio >= 1.80 (EventEngine) a gRPC channel that lives across
fork() breaks: continuous `Kick Failure (eventfd_write: Bad file
descriptor)` stderr spam and a racy silent deadlock of forked workers
inside gRPC's own at-fork handlers; grpcio 1.83.0 removed the last
legacy-poller opt-out (grpc/grpc#42828; upstream reports grpc/grpc#43055
and grpc/grpc#43062, both open and unfixed).
- `sw-python run -p gunicorn`: the master now only installs
instrumentation (start_prefork_master) and arms the fork hooks;
queues, reporter threads and the gRPC channel are created in each
forked worker only (the model uWSGI has always used). The master no
longer registers as a service instance.
- New agent.started() guard: instrumented code no-ops (NoopSpan,
dropped logs/meters) in a process whose reporters are not active,
protecting `gunicorn --preload` app imports in the master.
- GRPC_POLL_STRATEGY is no longer set; the grpcio floor is raised to
>= 1.83 (generated stubs already require it at import) and codegen
grpcio-tools is pinned in lockstep so future wheels cannot silently
raise the runtime floor.
- Gunicorn prefork + SW_AGENT_ASYNCIO_ENHANCEMENT is rejected instead
of starting an unsafe pre-fork agent.
- New plugin tests: sw_gunicorn (full consumer -> gunicorn-provider
trace with CrossProcess ref, exact worker-boot-count and
no-Kick-Failure log assertions, async-rejection case) and
sw_fork_support (explicit os.fork() with a continuous cross-fork
trace via SW_AGENT_EXPERIMENTAL_FORK_SUPPORT).
Validated in Docker on grpcio 1.80.0 / 1.82.1 / 1.83.0 x 8-10 boots
each: zero error spam, zero worker deadlocks, all segments delivered;
fd-level inspection confirms the master holds no collector connection
while each worker holds exactly one.
Fixesapache/skywalking#13958
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| agent_collector_heartbeat_period | SW_AGENT_COLLECTOR_HEARTBEAT_PERIOD | <class 'int'> | 30 | The agent will exchange heartbeat message with SkyWalking OAP backend every `period` seconds |
39
39
| agent_collector_properties_report_period_factor | SW_AGENT_COLLECTOR_PROPERTIES_REPORT_PERIOD_FACTOR | <class 'int'> | 10 | The agent will report service instance properties every `factor * heartbeat period` seconds default: 10*30 = 300 seconds |
40
40
| agent_instance_properties_json | SW_AGENT_INSTANCE_PROPERTIES_JSON | <class 'str'> || A custom JSON string to be reported as service instance properties, e.g. `{"key": "value"}`|
41
-
| agent_experimental_fork_support | SW_AGENT_EXPERIMENTAL_FORK_SUPPORT | <class 'bool'> | False | The agent will restart itself in any os.fork()-ed child process. Important Note: it's not suitable for short-lived processes as each one will create a new instance in SkyWalking dashboard in format of `service_instance-child(pid)`. This feature may not work when a precise combination of gRPC + Python 3.7 + subprocess (not fork) is used together. The agent will output a warning log when using on Python 3.7 for such a reason. |
41
+
| agent_experimental_fork_support | SW_AGENT_EXPERIMENTAL_FORK_SUPPORT | <class 'bool'> | False | The agent will restart itself in any os.fork()-ed child process. Important Note: it's not suitable for short-lived processes as each one will create a new instance in SkyWalking dashboard in format of `service_instance-child(pid)`. When the sw-python CLI detects a pre-forking server (Gunicorn), only worker processes run a full agent; the master installs instrumentation only. |
42
42
| agent_queue_timeout | SW_AGENT_QUEUE_TIMEOUT | <class 'int'> | 1 | DANGEROUS - This option controls the interval of each bulk report from telemetry data queues Do not modify unless you have evaluated its impact given your service load. |
43
43
| agent_asyncio_enhancement | SW_AGENT_ASYNCIO_ENHANCEMENT | <class 'bool'> | False | Replace the threads to asyncio coroutines to report telemetry data to the OAP. This option is experimental and may not work as expected. |
Copy file name to clipboardExpand all lines: docs/en/setup/faq/How-to-use-with-gunicorn.md
+17-6Lines changed: 17 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -14,7 +14,7 @@ serves requests.
14
14
15
15
> Note: Python 3.10+ is required. Earlier versions (3.7-3.9) are no longer supported.
16
16
17
-
**TL;DR:** specify `-p` or `--prefork` in `sw-python run -p` and all Gunicorn workers and master will get their own working agent.
17
+
**TL;DR:** specify `-p` or `--prefork` in `sw-python run -p` and every Gunicorn worker will get its own working agent (the master is instrumented only and runs no agent).
18
18
19
19
**Important:** if the call to gunicorn is prefixed with other commands, this approach will fail
20
20
since agent currently looks for the command line input at index 0 for safety as an experimental feature.
Copy file name to clipboardExpand all lines: docs/en/setup/faq/How-to-use-with-uwsgi.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -13,8 +13,8 @@ Some of the original discussion can be found here:
13
13
> You can always fall back to the manual approach.
14
14
> (although it's also possible to pass postfork hook without changing code, which is essentially how sw-python is implemented)
15
15
16
-
> Limitation: regardless of the approach used, uWSGI master process cannot be safely monitored. Since it doesn't take any requests, it is generally acceptable.
17
-
> Alternatively, you could switch to Gunicorn, where its master process can be monitored properly along with all child workers.
16
+
> Limitation: regardless of the approach used, the uWSGI master process is not monitored. Since it doesn't take any requests, it is generally acceptable.
17
+
> Gunicorn behaves the same way — only worker processes are monitored, which is the fork-safe design.
18
18
19
19
**Important**: The `--enable-threads` and `--master` option must be given to allow the usage of post_fork hooks and threading in workers.
20
20
In the `sw-python` CLI, these two options will be automatically injected for you in addition to the post_fork hook.
0 commit comments