You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
feat: native gRPC multi-backend failover via pick_first
One channel for the process lifetime with C-core pick_first over
comma-separated collector addresses (Node native failover analogue).
Single address stays plain host:port; multi uses ipv4:/ipv6: (mixed
families via IPv4-mapped). Multi hostnames expand once at channel build.
pick_first shuffleAddressList is on; target / default_authority stay in
config order.
Channel options: HTTP proxy off, no keepalive, reconnect backoff capped
at 30s, service_config retries only reportInstanceProperties. Skip
reports until READY. Unary and sync streaming RPCs use a deadline (10s
floor, always > queue window); aio client-streaming collect omits
timeout because generators await empty queues. Failed send batches are
counted as drops. Instance properties errors do not block keepAlive.
Replacing a protocol closes the previous channel (sync close; aio await
aclose on the agent loop). Timed shutdown flush so atexit cannot hang.
Includes unit coverage and a multi-OAP gRPC failover E2E case.
| agent_collector_backend_services | SW_AGENT_COLLECTOR_BACKEND_SERVICES | <class 'str'> | oap_host:oap_port| The backend OAP server address, 11800 is default OAP gRPC port, 12800 is HTTP, Kafka ignores this option and uses kafka_bootstrap_servers option. **This option should be changed accordingly with selected protocol**|
20
+
| agent_collector_backend_services | SW_AGENT_COLLECTOR_BACKEND_SERVICES | <class 'str'> | oap_host:oap_port| The backend OAP server address(es). 11800 is default OAP gRPC port, 12800 is HTTP, Kafka ignores this option and uses kafka_bootstrap_servers option. For gRPC, a comma-separated list enables native pick_first failover (per-process shuffle of the preferred backend; :authority stays the first configured endpoint). See Intrusive.md for encoding, DNS, TLS authority, proxy, and READY-gate details. **This option should be changed accordingly with selected protocol**|
21
21
| agent_protocol | SW_AGENT_PROTOCOL | <class 'str'> | grpc | The protocol to communicate with the backend OAP, `http`, `grpc` or `kafka`, **we highly suggest using `grpc` in production as it's well optimized than `http`**. The `kafka` protocol provides an alternative way to submit data to the backend. |
22
22
| agent_name | SW_AGENT_NAME | <class 'str'> | Python Service Name | The name of your awesome Python service |
23
23
| agent_instance_name | SW_AGENT_INSTANCE_NAME | <class 'str'> | str(uuid.uuid1()).replace('-', '') | The name of this particular awesome Python service instance |
| kafka_topic_log | SW_KAFKA_TOPIC_LOG | <class 'str'> | skywalking-logs | Specifying Kafka topic name for Log data, this should be in sync with OAP |
30
30
| kafka_topic_meter | SW_KAFKA_TOPIC_METER | <class 'str'> | skywalking-meters | Specifying Kafka topic name for Meter data, this should be in sync with OAP |
31
31
| kafka_reporter_custom_configurations | SW_KAFKA_REPORTER_CUSTOM_CONFIGURATIONS | <class 'str'> || The configs to init KafkaProducer, supports the basic arguments (whose type is either `str`, `bool`, or `int`) listed [here](https://kafka-python.readthedocs.io/en/master/apidoc/KafkaProducer.html#kafka.KafkaProducer) This config only works from env variables, each one should be passed in `SW_KAFKA_REPORTER_CONFIG_<KEY_NAME>`|
32
-
| agent_force_tls | SW_AGENT_FORCE_TLS | <class 'bool'> | False | Use TLS for communication with SkyWalking OAP (no cert required) |
32
+
| agent_force_tls | SW_AGENT_FORCE_TLS | <class 'bool'> | False | Use TLS for gRPC/HTTP with the OAP (no client cert required). For gRPC, ensure the server certificate SAN matches the first usable backend in agent_collector_backend_services (used as grpc.default_authority).|
33
33
| agent_authentication | SW_AGENT_AUTHENTICATION | <class 'str'> || The authentication token to verify that the agent is trusted by the backend OAP, as for how to configure the backend, refer to [the yaml](https://github.com/apache/skywalking/blob/4f0f39ffccdc9b41049903cc540b8904f7c9728e/oap-server/server-bootstrap/src/main/resources/application.yml#L155-L158). |
34
34
| agent_logging_level | SW_AGENT_LOGGING_LEVEL | <class 'str'> | INFO | The level of agent self-logs, could be one of `CRITICAL`, `FATAL`, `ERROR`, `WARN`(`WARNING`), `INFO`, `DEBUG`. Please turn on debug if an issue is encountered to find out what's going on |
35
35
### Agent Core Danger Zone
36
36
| Configuration | Environment Variable | Type | Default Value | Description |
| agent_collector_heartbeat_period | SW_AGENT_COLLECTOR_HEARTBEAT_PERIOD | <class 'int'> | 30 | The agent will exchange heartbeat message with SkyWalking OAP backend every `period` seconds |
39
-
| agent_collector_properties_report_period_factor | SW_AGENT_COLLECTOR_PROPERTIES_REPORT_PERIOD_FACTOR | <class 'int'> | 10 | The agent will report service instance properties every `factor * heartbeat period` seconds default: 10*30 = 300 seconds |
39
+
| agent_collector_properties_report_period_factor | SW_AGENT_COLLECTOR_PROPERTIES_REPORT_PERIOD_FACTOR | <class 'int'> | 10 | The agent will report service instance properties every `factor * heartbeat period` seconds default: 10*30 = 300 seconds (Java/Node cadence). Also covers gRPC pick_first silent backend switches that stay READY without a disconnect event. |
40
40
| agent_instance_properties_json | SW_AGENT_INSTANCE_PROPERTIES_JSON | <class 'str'> || A custom JSON string to be reported as service instance properties, e.g. `{"key": "value"}`|
41
41
| agent_experimental_fork_support | SW_AGENT_EXPERIMENTAL_FORK_SUPPORT | <class 'bool'> | False | The agent will restart itself in any os.fork()-ed child process. Important Note: it's not suitable for short-lived processes as each one will create a new instance in SkyWalking dashboard in format of `service_instance-child(pid)`. When the sw-python CLI detects a pre-forking server (Gunicorn), only worker processes run a full agent; the master installs instrumentation only. |
42
42
| agent_queue_timeout | SW_AGENT_QUEUE_TIMEOUT | <class 'int'> | 1 | DANGEROUS - This option controls the interval of each bulk report from telemetry data queues Do not modify unless you have evaluated its impact given your service load. |
Pass a comma-separated list. The agent opens **one** gRPC channel and lets C-core `pick_first` fail over (same idea as Node `sw-static`). Each process shuffles the preferred backend at channel build; `:authority` / TLS SNI still use the **first configured** endpoint.
- Mixed IPv4/IPv6 stays in one list; IPv4 is encoded as IPv4-mapped IPv6 for grpcio so `pick_first` can try both families.
45
+
- Multi-hostname lists are DNS-expanded once at channel build (about 5s lookup budget per name); there is no periodic re-resolve — prefer a single address or stable IPs when DNS changes.
46
+
- Channel `:authority` / TLS SAN uses `grpc.default_authority` = the first configured endpoint (before shuffle). With `agent_force_tls`, every backend cert must cover that authority.
47
+
- Reporters wait until the channel is READY; non-READY skips the RPC rather than failing fast into a black hole.
48
+
- After a silent backend switch that stays READY, instance properties are re-reported on the normal properties period so the new OAP learns the instance.
49
+
- Reconnect backoff caps at 30s.
50
+
51
+
#### gRPC HTTP proxy (behavior change)
52
+
53
+
`grpc.enable_http_proxy=0` is set on **every** gRPC channel, including a single-address config. Host `http_proxy` / `https_proxy` / `no_proxy` are ignored for OAP traffic.
54
+
55
+
If you currently reach OAP only through an HTTP CONNECT proxy, upgrading this agent will lose that path. The agent disables the proxy so application proxy env vars cannot silently black-hole telemetry, and because HTTP CONNECT happens before name resolution — it cannot express a comma-separated backend list.
56
+
57
+
#### Best-effort reporting and buffers
58
+
59
+
Send failures are discarded and never retried (a failed batch is dropped). On process shutdown the agent flushes only while the channel is READY, with a short time budget, then **abandons** whatever is still queued.
60
+
61
+
When a reporter queue is full (`SW_AGENT_TRACE_REPORTER_MAX_BUFFER_SIZE` / log / meter / snapshot equivalents), new items are dropped. The agent logs drops at most once per 30 seconds, with the increment since the last line and the process total. Raise the buffer if you routinely see these under load; a full queue during an outage is expected because the READY gate holds data until the backend returns.
62
+
30
63
### Report data via HTTP protocol
31
64
32
65
However, if you want to use HTTP protocol to report data, configure `agent_collector_backend_services`
0 commit comments