the network is not a wire

In the script's output, stage A and stage B both have p99=0 ms. The combined path has p99=100 ms. I supplied all 1,000 synthetic request timings to the reproduction, so I can see exactly where the summaries lost information: they kept the delay and discarded the identity of the slow requests. Before explaining that result, I want to follow the places where a request can wait.

Here is a FIFO with 125 MB ahead of our small packet and a 100 Mbps link to drain it. The units need converting before I divide: 100 Mbps / 8 = 12.5 MB/s, in decimal bytes. At that rate, clearing the backlog takes ten seconds. Put another way, 125 MB occupies 1,000 megabits of the link's capacity. It could be busy for the entire wait, making the throughput graph look excellent while the shell command or call behind the backlog stalls. I need the bytes to leave faster to shorten that wait. Giving them a bigger buffer would only allow a longer backlog.

after send() returns

Start the request's path at send(). A successful return can mean the bytes are still in the sending machine's socket send buffer. They may wait again at the NIC's transmit ring, then in switches and routers. At the far end, the receiving application has to be scheduled even after the bytes reach its network stack. I would put those waits on the drawing alongside the physical links.

Hold the FIFO's departure rate fixed and divide the bytes ahead by that rate for the approximate wait. Arrive just before a burst and the queue may be empty; arrive just after and those bytes are now ahead of you. That split can vanish in an average. It makes queueing worth checking when p99 rises, alongside retransmissions and time waiting for the application to run.

waiting for a congestion signal

From the sender's end, most of those queues are out of sight. It has acknowledgments for delivery and timing, plus loss or ECN marks as possible congestion signals. It rarely knows every link's capacity. And while it waits for feedback, other senders change how much capacity is available. They are making their own decisions from delayed feedback too.

For Reno, picture the sender with a window of 64 segments in congestion avoidance. Give it an RTT of successful delivery and it can grow to about 65; another, about 66. Then three duplicate acknowledgments arrive. The loss response roughly halves the window, so the climb starts again from lower down. A timeout is a more severe setback. That repeated climb and cut gives the diagram its sawtooth. Jacobson's 1988 paper traces the feedback problem that led to congestion collapse.

For CUBIC, start a loss event with a window of 100 segments. Its usual decrease would leave 70, or 70%, after which growth follows a cubic function of elapsed time since the congestion event. RFC 9438 specifies the rule. Follow it or Reno while our router keeps accepting packets: neither sees an overflow signal yet. They can continue probing upward while bytes are already piling up in that router. Changing how the window grows has left the signal arriving too late.

Let the burst drain and the buffer has done useful work. Let the backlog persist and we have bufferbloat. Extra buffer space can move the overflow signal further away, so the sender leaves more bytes waiting before it learns to back off. Gettys and Nichols describe this failure in Bufferbloat: Dark Buffers in the Internet. Our ten-second FIFO makes its cost easy to see: all that latency while the throughput graph still looks busy.

CoDel looks for a queue that stays

Six milliseconds of waiting by itself tells me little. Perhaps this packet arrived just after a burst and the next few will find an empty queue. CoDel watches the least waiting time among recent departures to distinguish that case. The defaults in RFC 8289 use 5 ms as the target. The other number, a 100 ms interval, is what keeps me from reading that target as an expiration date on each packet. If even the least-delayed departures stay above the target through that interval, the backlog has persisted. A burst has had time to clear before we act on a standing queue.

CoDel can now send the warning as a drop or ECN mark while the buffer still has space. Add fq_codel and we can also change which flow gets the next turn. With a sparse SSH flow in its own queue, packets need not sit behind a bulk transfer's entire backlog. RFC 8290 describes that treatment of sparse flows. I would first locate the queue I mean to fix. Enabling fq_codel on a host changes that host's scheduling; it cannot pull bytes out of a backlog already waiting in another router.

250 kB in flight, another 1 MB waiting

For the same link, take an otherwise unqueued RTT of 20 ms. During that time it sends about 250 kB: 12.5 MB/s × 0.020 s = 0.25 MB. That is the bandwidth-delay product, our estimate of what needs to be in flight to keep this path busy. What if I add 1 MB of queue at the bottleneck? At 12.5 MB/s those extra bytes take another 80 ms to clear, without raising the delivery rate.

BBR needs estimates for the two inputs we just multiplied: delivery rate and baseline RTT. It uses them to find an operating point and paces packet departures instead of releasing everything at once. Cardwell and colleagues' 2016 paper describes the original model. To find spare bandwidth, the sender has to try sending faster. To observe baseline RTT, it needs a measurement without its own standing queue in the way. Those probes move its sending rate, which is why I would not read the diagram's flat BBR reference as a trace of the controller running.

A bandwidth probe can itself build a queue. Put competing flows on the path and the estimates get harder to interpret. Later BBR versions changed loss response and behavior alongside other flows, so a useful comparison needs the version and the competing traffic beside its results. I would look at delivered rate together with latency under load. Adjusting the offered load gives the sender some control, but it reserves no capacity along the path from which to guarantee a request's deadline.

p99 needs the requests to stay paired

There is a probability calculation I want to keep separate from the p99 trace. Give each of ten independent hops a 1% chance of being slow. The chance of avoiding every slow hop is 0.99^10, so the chance of meeting at least one is 1 − 0.99^10 ≈ 9.6%. We have counted encounters with a slow hop. We still need the delay that "slow" adds to calculate request latency. Shared congestion could also make the independence assumption wrong.

Run the script and inspect the request IDs before looking at the percentiles. In stage A, requests 1–6 each take 100 ms. Stage B delays a different set, 7–12, by the same amount. All other times are zero. That puts 994 zeros at the beginning of either sorted stage array: 99.4% of the 1,000 samples. Nearest-rank p99 asks for sample 990, which is still zero in both.

Adding the stages with zip keeps each request with its own timings. There are now twelve slow sums and only 988 zeros. Look for sample 990 in that sorted array: it has to be one of the 100 ms requests. The printed end-to-end p99 is therefore 100 ms. Both stage summaries had printed 0 ms. I supplied these synthetic times, with exact request identities, so there are no real clocks to align and no trace to reconstruct. Reducing each stage to a percentile was enough to lose the information.

Dispatch adds another place for a request to wait. A random choice might put it on a busy server while an idle one is available; sampling two queues and choosing the shorter makes that less likely. In Mitzenmacher's power of two choices work, I can compare those choices in a static setup: n jobs enter n initially empty queues. The maximum load with one random choice has the bound O(log n / log log n), with high probability. Allowing two choices improves that bound to O(log log n). I would try sampling in a server design, then check the things this static setup left out. Jobs depart, their costs differ, and a queue report may already be stale when the dispatcher acts on it.

To investigate a real p99, I would first compare idle RTT with RTT during a bulk transfer. Rising RTT with steady server processing would send me along the network path looking for queued bytes. If RTT stayed flat, I would follow a growing application span into the server queues and connection pools instead. Queue-depth and drop counters could help test either explanation. Then I could change the controller's offered traffic or the scheduler's choice of the next packet and repeat the experiment. I would keep the traffic the same before and after, recording delivered rate and latency under load. The request timings would stay paired until that comparison was done.