Skip to content

Commit 7b5310a

Browse files
wu-shengclaude
andcommitted
docs: honest fork-safety guidance; test gunicorn --preload; review fixes
Review follow-ups: - Explicit os.fork() with a live gRPC channel is documented (and warned at runtime, outside the Gunicorn prefork path) as unreliable on grpcio >= 1.80 — background reporters enter gRPC independent of requests, upstream races grpc/grpc#43055/#43062 remain open; forking applications are directed to SW_AGENT_PROTOCOL=http/kafka. Gunicorn via `sw-python run -p` is called out as the supported channel-after- fork model. - The asyncio enhancement + prefork incompatibility is now documented in the Gunicorn FAQ, AsyncEnhancement.md and the config reference. - sw_grpc plugin note states the effective grpcio floor (>= 1.83 via the agent package) alongside the 1.* instrumentation range. - __fini is registered once per process lineage (atexit registrations are fork-inherited), mirroring the at-fork hook guard. - The sw_gunicorn provider now runs with --preload and performs an instrumented call at module import, covering the agent.started() no-op guard in the instrumentation-only master. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent bbe2531 commit 7b5310a

10 files changed

Lines changed: 47 additions & 8 deletions

File tree

docs/en/setup/CLI.md

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -71,9 +71,13 @@ Note that `sw-python` also work with spawned subprocess (os.exec*/subprocess) as
7171

7272
Additionally, `sw-python` started agent works well with `os.fork` when your application forks workers,
7373
as long as the `SW_AGENT_EXPERIMENTAL_FORK_SUPPORT` is turned on. (It will be automatically turned on when gunicorn is detected)
74-
Avoid calling `os.fork()` while the agent is actively talking to the collector (e.g. immediately at startup, during
75-
registration): a fork during an in-flight gRPC call can trip gRPC's own at-fork handling (see grpc/grpc#43055) and
76-
hang the child. Forking a moment after startup, or between requests, is safe.
74+
75+
**Important**: with the default gRPC reporter, explicit `os.fork()` is NOT reliable on grpcio >= 1.80: the agent's
76+
background reporters enter gRPC at any time (heartbeat, segment flush) independent of application requests, and a
77+
fork while a channel is live is subject to open upstream races (grpc/grpc#43055, grpc/grpc#43062) that can silently
78+
break reporting in either process. Applications that fork should use `SW_AGENT_PROTOCOL=http` (or `kafka`).
79+
Gunicorn via `sw-python run -p` is NOT affected: there the agent creates its channels only after the fork,
80+
which is gRPC's supported model.
7781

7882
## Configuring the agent
7983

docs/en/setup/Configuration.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ export SW_AGENT_YourConfiguration=YourValue
4040
| agent_instance_properties_json | SW_AGENT_INSTANCE_PROPERTIES_JSON | <class 'str'> | | A custom JSON string to be reported as service instance properties, e.g. `{"key": "value"}` |
4141
| agent_experimental_fork_support | SW_AGENT_EXPERIMENTAL_FORK_SUPPORT | <class 'bool'> | False | The agent will restart itself in any os.fork()-ed child process. Important Note: it's not suitable for short-lived processes as each one will create a new instance in SkyWalking dashboard in format of `service_instance-child(pid)`. When the sw-python CLI detects a pre-forking server (Gunicorn), only worker processes run a full agent; the master installs instrumentation only. |
4242
| agent_queue_timeout | SW_AGENT_QUEUE_TIMEOUT | <class 'int'> | 1 | DANGEROUS - This option controls the interval of each bulk report from telemetry data queues Do not modify unless you have evaluated its impact given your service load. |
43-
| agent_asyncio_enhancement | SW_AGENT_ASYNCIO_ENHANCEMENT | <class 'bool'> | False | Replace the threads to asyncio coroutines to report telemetry data to the OAP. This option is experimental and may not work as expected. |
43+
| agent_asyncio_enhancement | SW_AGENT_ASYNCIO_ENHANCEMENT | <class 'bool'> | False | Replace the threads to asyncio coroutines to report telemetry data to the OAP. This option is experimental and may not work as expected. Not compatible with pre-forking servers (`sw-python run -p`): the agent refuses to start under a Gunicorn master. |
4444
### SW_PYTHON Auto Instrumentation CLI
4545
| Configuration | Environment Variable | Type | Default Value | Description |
4646
| :------------ | :------------ | :------------ | :------------ | :------------ |

docs/en/setup/Plugins.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,8 @@ in SkyWalking currently. Celery clients can use whatever protocol they want.
6060
Hug is believed to be abandoned project, use this plugin with a bit more caution.
6161
Instead of Hug, plugin test should move to test actual Falcon.
6262
- Falcon 3.x/4.x plugin. For legacy hug-based instrumentation, see sw_falcon.
63+
- The agent package itself depends on grpcio >= 1.83, which is therefore the
64+
effective minimum version of the instrumented library as well.
6365
- The Neo4j plugin integrates neo4j python driver 5.x.x versions which
6466
support both Neo4j 5 and 4.4 DBMS.
6567
- Sanic 21.9+ plugin using signal listeners.

docs/en/setup/advanced/AsyncEnhancement.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,10 @@ Since `1.1.0`, the Python agent supports asynchronous reporting of ALL telemetry
66
export SW_AGENT_ASYNCIO_ENHANCEMENT=true
77
```
88

9+
> Limitation: this option is incompatible with pre-forking servers via `sw-python run -p` (e.g. Gunicorn) —
10+
> the asyncio agent has no fork support, so the agent refuses to start and the application runs without
11+
> observability. See the [Gunicorn FAQ](../faq/How-to-use-with-gunicorn.md).
12+
913
## Why we need this feature
1014

1115
Before version `1.1.0`, SkyWalking Python agent had only an implementation with the Threading module to provide data reporters. Yet with the growth of the Python agent, it is now fully capable and requires more resources than when only tracing was supported (we start many threads and gRPC itself creates even more threads when streaming).

docs/en/setup/faq/How-to-use-with-gunicorn.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,12 @@ and can silently hang workers. Tracing generally keeps working — the errors co
4646
engine and are unrelated to the OAP version. Workarounds on old agents: pin `grpcio<1.80` or use `SW_AGENT_PROTOCOL=http`.
4747
Fixed agent versions require `grpcio >= 1.83` and never create a gRPC channel in the master.
4848

49+
### Incompatible with the asyncio enhancement
50+
51+
`SW_AGENT_ASYNCIO_ENHANCEMENT=true` is incompatible with `sw-python run -p gunicorn`: the asyncio agent has no
52+
fork support, so the agent refuses to start (an error is logged) and the application serves WITHOUT observability.
53+
Remove the asyncio enhancement option, or run Gunicorn without `-p`.
54+
4955
## Manual Approach (only use when sw-python doesn't work)
5056

5157
**Limitation**: Using normal postfork hook will not add observability to the master process.

skywalking/agent/__init__.py

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -115,6 +115,7 @@ def __init__(self):
115115
# True only after __bootstrap() in the current process; stays False in a pre-fork master
116116
self.__reporting: bool = False
117117
self.__at_fork_registered: bool = False
118+
self.__fini_registered: bool = False
118119

119120
def __bootstrap(self):
120121
# when forking, already instrumented modules must not be instrumented again
@@ -273,6 +274,12 @@ def start(self) -> None:
273274
# see https://github.com/apache/skywalking/issues/13958
274275
os.environ['GRPC_ENABLE_FORK_SUPPORT'] = 'true' # must precede `import grpc`
275276

277+
if not os.getenv('prefork'): # Gunicorn prefork creates channels only after fork() and is safe
278+
logger.warning('Explicit os.fork() with a live gRPC channel is unreliable on '
279+
'grpcio >= 1.80 (see grpc/grpc#43055) and may silently break '
280+
'reporting in either process; prefer SW_AGENT_PROTOCOL=http or '
281+
'kafka for forking applications.')
282+
276283
if not self.__started:
277284
# if not already started, start the agent
278285
logger.info(f'SkyWalking sync agent instance {config.agent_instance_name} starting in pid-{os.getpid()}.')
@@ -305,7 +312,11 @@ def start(self) -> None:
305312

306313
self.__bootstrap() # calls init_threading
307314

308-
atexit.register(self.__fini)
315+
# atexit registrations are fork-inherited; register once per lineage so a
316+
# fork-restarted child does not stack a duplicate __fini
317+
if not self.__fini_registered:
318+
self.__fini_registered = True
319+
atexit.register(self.__fini)
309320

310321
if config.agent_experimental_fork_support:
311322
self.__register_fork_hooks()
@@ -387,6 +398,7 @@ def stop(self) -> None:
387398
Stops the agent and reset the started flag.
388399
"""
389400
atexit.unregister(self.__fini)
401+
self.__fini_registered = False
390402
self.__fini()
391403
self.__reporting = False
392404
self.__started = False

skywalking/config.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -102,7 +102,8 @@
102102
# Do not modify unless you have evaluated its impact given your service load.
103103
agent_queue_timeout: int = int(os.getenv('SW_AGENT_QUEUE_TIMEOUT', '1'))
104104
# Replace the threads to asyncio coroutines to report telemetry data to the OAP.
105-
# This option is experimental and may not work as expected.
105+
# This option is experimental and may not work as expected. Not compatible with pre-forking
106+
# servers (`sw-python run -p`): the agent refuses to start under a Gunicorn master.
106107
agent_asyncio_enhancement: bool = os.getenv('SW_AGENT_ASYNCIO_ENHANCEMENT', '').lower() == 'true'
107108

108109
# BEGIN: SW_PYTHON Auto Instrumentation CLI

skywalking/plugins/sw_grpc.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,8 @@
2727

2828
link_vector = ['https://grpc.io/docs/languages/python']
2929
support_matrix = {'grpcio': {'>=3.8': ['1.*']}}
30-
note = """"""
30+
note = """The agent package itself depends on grpcio >= 1.83, which is therefore the
31+
effective minimum version of the instrumented library as well."""
3132

3233

3334
def _get_factory_and_method(rpc_handler: Any) -> Tuple[Callable[..., Any], Callable[..., Any]]:

tests/plugin/web/sw_gunicorn/docker-compose.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ services:
2929
file: ../../docker-compose.base.yml
3030
volumes:
3131
- .:/app
32-
command: ['bash', '-c', 'pip install flask gunicorn && pip install -r /app/requirements.txt && cd /app/services && sw-python run -p gunicorn provider:app --workers 2 --threads 2 --bind 0.0.0.0:9091']
32+
command: ['bash', '-c', 'pip install flask gunicorn && pip install -r /app/requirements.txt && cd /app/services && sw-python run -p gunicorn provider:app --preload --workers 2 --threads 2 --bind 0.0.0.0:9091']
3333
depends_on:
3434
collector:
3535
condition: service_healthy

tests/plugin/web/sw_gunicorn/services/provider.py

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,8 +14,17 @@
1414
# See the License for the specific language governing permissions and
1515
# limitations under the License.
1616
#
17+
import requests
1718
from flask import Flask, jsonify
1819

20+
try:
21+
# under gunicorn --preload this module imports in the PRE-FORK master, where the
22+
# agent is instrumentation-only: this instrumented call must silently noop
23+
# (NoopSpan via agent.started() guard) instead of crashing the master
24+
requests.get('http://collector:12800/receiveData', timeout=5)
25+
except Exception: # noqa
26+
pass
27+
1928
app = Flask(__name__)
2029

2130

0 commit comments

Comments
 (0)