Skip to content

Commit 0e7fcc4

Browse files
testillanoEduardo Ramos Testillano (eramedu)
authored andcommitted
Add Process Resources section to Grafana dashboard
Add row with panels for memory usage (RSS/virtual), CPU usage, file descriptors and threads monitoring. Also clarify rejection scenario (503) and how to overcome effective rate degradation in high load testing.
1 parent ea0102b commit 0e7fcc4

4 files changed

Lines changed: 460 additions & 4 deletions

File tree

README.md

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -639,6 +639,22 @@ Also, reports are generated as markdown files under the profile's `reports/` sub
639639
* Data storage is disabled in the script by default to prevent memory from growing and improve server response times (remember that storage shall be kept when provisions require data persistence).
640640
* In general, even with high traffic rates, you could get sneaky snapshots just enabling and then quickly disabling data storage, for example using [function helpers](#Helper-functions): `server_data_configuration --keep-all && server_data_configuration --discard-all`
641641

642+
#### Handling 503 rejections in client mode
643+
644+
When using h2agent as a traffic client against a server under congestion, 503 responses indicate the server cannot sustain the requested rate.
645+
646+
**Breaking the chain on rejection** (detecting the problem):
647+
648+
* **`expectedResponseStatusCode`**: set the expected value (e.g., `201`). If the actual response differs, the chain breaks immediately. Simple and deterministic.
649+
* **`onResponseTransform` with filters**: read `response.statusCode` as source, apply conditional logic via filters. On mismatch, use `onFilterFail` to set an error `outState`, write diagnostic vaults, or trigger alternative flows. More flexible than the binary kill-switch above.
650+
* **Omit `expectedResponseStatusCode`**: the chain continues regardless of status code. Useful when you want to collect all events for post-hoc analysis without interrupting the flow.
651+
652+
**Managing load under congestion** (mitigating the problem):
653+
654+
* **External rate controller**: monitor the `h2agent_traffic_client_observed_responses_received_counter{status_code="503"}` metric and adjust the trigger rate dynamically (e.g., a PID loop that reduces CPS when rejection rate exceeds a threshold, or compensates effective load by increasing rate to offset rejections).
655+
* **Tiered rate ramp**: start at low rate and increase stepwise, measuring sustainable throughput at each plateau. Stop when 503 rate exceeds acceptable threshold -- this measures Engineering Capacity (EC).
656+
* **Long ramp-up**: increase rate gradually over time to let the server warm up and stabilize, reducing transient rejections during rate transitions.
657+
642658

643659

644660
So you may start the process, again, natively or using docker:

build-native.sh

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -17,10 +17,10 @@ boost_ver=1.84.0 # safer to have this version (https://github.com/nghttp2/nghttp
1717
ert_nghttp2_ver=v1.2.9 # to download nghttp2 patches (this must be aligned with previous: nghttp2 & nghttp2-asio & boost)
1818
ert_logger_ver=v1.1.1
1919
ert_queuedispatcher_ver=v1.0.4
20-
jupp0r_prometheuscpp_ver=v0.13.0
21-
civetweb_civetweb_ver=v1.14
22-
ert_metrics_ver=v1.1.1
23-
ert_http2comm_ver=v2.2.15
20+
jupp0r_prometheuscpp_ver=v1.3.0
21+
civetweb_civetweb_ver=v1.16
22+
ert_metrics_ver=v1.2.0
23+
ert_http2comm_ver=v2.3.0
2424
nlohmann_json_ver=$(grep ^nlohmann_json_ver__dflt= ${project_root_dir}/build.sh | cut -d= -f2)
2525
pboettch_jsonschemavalidator_ver=$(grep ^pboettch_jsonschemavalidator_ver__dflt= ${project_root_dir}/build.sh | cut -d= -f2)
2626
google_test_ver=$(grep ^google_test_ver__dflt= ${project_root_dir}/build.sh | cut -d= -f2)

src/main.cpp

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1083,6 +1083,7 @@ ChatGPT: https://github.com/testillano/h2agent/blob/master/README.md#ques
10831083
std::cerr << currentDateTime() << ": Initialization error in prometheus interface (" << bind_address_port_prometheus_exposer << "). Exiting ..." << '\n';
10841084
myExit(EXIT_FAILURE);
10851085
}
1086+
myMetrics->enableProcessMetrics(application_name);
10861087
}
10871088

10881089
// FileManager/SafeFile metrics

0 commit comments

Comments
 (0)