The Load Generator Is Part Of The Benchmark

A benchmark is invalid when the generator, network path, client coordination, or measurement pipeline saturates before the system under test.

When designing a benchmark I treat the generator and the measurement pipeline as first-class components of the system under test. If any of those earlier-stage pieces hit their limits you get a result about the test harness, not the target. The practical cost of this mistake is wasted time, false conclusions about capacity, and wrong engineering decisions.

Generator Saturation Is A Measurement Failure

Generator saturation shows up as a ceiling on offered load, unexpected latency plateaus, or a collapse in percentiles that don’t match resource signals. The generator can fail in many ways: CPU and scheduler saturation, socket or ephemeral-port exhaustion, TCP connection churn, single-threaded event-loop limits, or simply the NIC/host hitting its I/O ceiling. The network path between generator and target can also cap throughput: firewalls, load balancers, software proxies, and switch buffers are all suspects.

Detecting generator saturation requires treating the generator as a system: measure its run-queue, per-thread CPU, socket table, NIC utilization, retransmission counters, and appropriate kernel queues. Don’t infer “target saturated” from latency alone — correlate it with generator-side telemetry and the network path.

Tradeoffs: investing in a more powerful generator cluster or smarter client code costs time and money. In many cases simpler client adapters (connection pooling, async IO) reduce the generator blast radius without buying new hardware.

Control The Workload Shape: Closed Versus Open Workload

The difference between closed versus open workload is a design constraint with operational consequences. A closed workload couples client concurrency to response time: clients send the next request only after the previous one completes. Open workloads push requests at an externally controlled rate regardless of responses.

Closed workloads hide queueing in the client. Under load, response times rise, client concurrency effectively shrinks, and offered load falls — the generator looks like it’s “self-throttling.” Open workloads place the offered load outside the client, which exposes the target’s ability to absorb work but requires the generator to sustain the rate.

Pick the shape that matches the real user model. When evaluating raw capacity prefer open workloads with independent offered-load validation. When validating end-to-end user experience, closed workloads are valid but you must record offered-load decay as a signal, not assume constant intensity.

Clock And Metric Alignment Prevents False Attribution

Clock and metric alignment is the plumbing that makes correlation trustworthy. Mismatched clocks, different aggregation windows, or collector batching behavior will move spikes across time and create phantom bottlenecks.

Use monotonic timestamps for durations, align aggregation windows across client, network, and server telemetry, and record exporter latency. Tag every metric with the source and a collection timestamp before any downstream buffering. If you must correlate traces and logs, include both monotonic and wall-clock timestamps so you can reconcile order even when NTP shifts.

Failure modes: relying on wall-clock timestamps from different hosts without checking skew; mixing meter export intervals (one source at 10s, another at 60s) that hide short saturations; and trusting a single histogram without knowing bucket reset semantics.

Independent Validation Of Offered Load Is Non-Negotiable

Independent validation of offered load means you do not trust the generator’s internal counters as the sole source of truth. Measure the load at least one other way: server-side request counters, packet captures, or network-level byte counters.

A sequence for independent validation:

• Observe client-reported offered QPS and concurrent requests.
• Query server-side request counters and compare windows to client reports.
• Capture TCP-level packets or use a mirrored NIC to count request packets if protocol allows lightweight counting.
• Check middleboxes and LB metrics for discrepancies.

If the client says it sent 100k requests but the server and network show fewer, the generator, network path, or a proxy is the limiting factor — not the target.

Checklist: Benchmark Validation And Acceptance Criteria

Use this checklist before trusting any capacity number:

• Baseline generator health: CPU run-queue, per-thread utilization, NIC transmit/receive, socket table occupancy.
• Network path check: switch/virtual switch buffers, load-balancer queue lengths, and firewall throughput metrics.
• Workload shape chosen: explicitly document closed versus open workload rationale.
• Offered-load cross-check: server counters, packet counts, and client counters all recorded for the same aligned windows (see clock and metric alignment).
• Collector pipeline sanity: exporter queues, retry/backpressure behavior, and observed exporter latencies.
• Dry run ramp: gradually increase offered load while observing whether offered load diverges from attempted rate.
• Acceptance criteria: recorded offered load matches target within an agreed tolerance and remains stable for the target stability window.
• Post-run audit: save raw traces, server counters, and generator telemetry to reproduce the inference path.

This checklist is a minimal validation harness. Expand it with your organization’s acceptance thresholds and rollback rules.

When This Advice Is Wrong (And What To Do Instead)

There are valid cases where the generator intentionally limits the load: microbenchmarking a single-threaded code path, testing client-side behavior under resource constraints, or simulating a low-volume production client. In those cases the generator is intentionally part of the test hypothesis — make that explicit in acceptance criteria and documentation.

If the goal is to exercise a target at scale then the generator must not be the limit. If you can’t provision a stronger generator, instrument the target to accept a smaller, well-understood workload and reframe the benchmark as a modeled projection rather than an absolute measurement.

Takeaway

A benchmark’s credibility depends on proving the offered load actually reached the target and that measurement timing is correct. Treat the generator, the network path, client coordination, and the measurement pipeline as components with their own failure modes. Use independent validation of offered load, enforce clock and metric alignment, and pick closed versus open workload deliberately. When the generator saturates, stop trusting the headline numbers: trace the signals, fix the bottleneck, or reframe the question.

If you want a short checklist adapted to your stack, contact me at /contact for the template and example telemetry queries.