Everyone I show watson to asks the same question: what’s the time to first token? How long after I ask does something appear on screen?
That’s a fair question, and it’s awkward for a tool that promises never to read content. A token is content. So I tried to infer it from nothing but packet timing and sizes, and set a pass mark before looking at the results. The first set of rules failed it.
The problem
Time to first byte is easy to measure and it doesn’t answer the question. On an AI stream, response headers and a small preamble can arrive long before the first token. The wait in between is what you actually feel.
Here’s a row from my log on November 7, from Claude Code:
| Time to first byte | 464 ms |
| Time to first token | 2.89 s |
| Silence between them | 2.42 s, classed as provider wait |
A dashboard showing TTFB would call that a fast response. It took six times longer than that to start answering.
There’s a second problem underneath. While a large request uploads, the server sends HTTP/2 control frames back. They look like a first byte. Before I fixed this, every large upload got split into 8 to 24 fake multiplexed interactions, so even TTFB was wrong on the requests that mattered most.
How watson solves it
TTFT comes from two rules, both working only from sizes and gaps.
Rule 1 finds the first byte. A WINDOW_UPDATE is one 35 B record under TLS 1.3, and TCP packs several into a segment. So during the request phase, a downstream segment that’s a multiple of 35 B and under 1000 B is control, and gets dropped. Any other segment under 256 B is held. If a segment of 256 B or more arrives within 50 ms of the last held one, the held segments were the start of the response. More upload, or 50 ms of quiet, means they were control. Retransmits are skipped.
Rule 2 finds the first token. Take the response’s opening arrival, meaning packets under 5 ms apart. If the next arrival comes within 750 ms, the stream is already flowing and the first token is the first byte. If 750 ms or more of silence follows, the model is thinking, the opening was only a preamble, and the first token is the next arrival.
That’s the whole method. I tested it with a harness that streamed questions to the Anthropic API, logged the real time of the first content_block_delta, and captured the traffic with tcpdump. The capture was replayed through the same rules. I varied the prompt size, adaptive thinking and new or reused connections.
The bar was fixed before any data came in: at least 50 questions, at least 90% within ±150 ms of the true first token, and no ping ever taken for a token.
| Round | Questions | Within ±150 ms | What happened |
|---|---|---|---|
| 1 | 26 | 77%, then 100% after fitting | Stopped early, used for fitting |
| 2 | 60 | 88%, then 100% after fitting | Held out for the first rules, which failed the bar |
| 3 | 7 | Ran out of API credit, discarded | |
| 3b | 60 | 98% (59 of 60) | Held out for the final rules |
Round 2 is why I bother with held-out data. The rules looked perfect on the round I’d fitted them to, then came in at 88% on questions they hadn’t seen. Scoring on the same data would have shipped a broken metric with a 100% next to it.
I should also be upfront about two changes around the final score. Round 3b first came out at 53 of 53, with 7 questions unseen because their 224 B requests fell under a 300 B threshold. I lowered that threshold to 160 B and replayed the same capture. Later I added the 1000 B cap in rule 1, because an MSS clamped to 1400 B, a multiple of 35 that’s common on VPNs, would otherwise drop every full-size response segment. It’s above every control segment I saw, so it changed no result, but both of these happened after scoring, and the decision record says so.
Then I checked it against real Claude Code rather than my harness. I asked three short questions, one “think hard” question, one tool question and one longer answer. The short ones had TTFT equal to TTFB, at 1.7 to 2.1 s. The thinking question had headers at 2.4 s, 2.7 s of silence and a first token at 5.0 s. Both matched my own sense of the wait, about 2 s and about 5 s. That’s a sanity check and a weak one, but I was glad it agreed.
Here’s the turn that row came from. The last line is the one above: 464 ms to the first byte, then the pink first-token wait in the timeline, then the first token at 2.89 s.
Troubleshooting steps
TTFT is reported and never scored, so it’s something you read yourself.
- Find the turn in
watson watchor on the page. The header looks likeTTFT 2.13s · max 5.01s, with(vs normal …)once the app has a baseline. The headline is the first streamed answer’s TTFT, so it matches the marker on the page. - Compare TTFT with TTFB. If they’re about equal, the app sent its headers when the first token was ready, and there’s no hidden thinking wait to explain.
- If TTFT is far above TTFB, look at the class of the wait.
provider-waitmeans the network looked healthy while nothing arrived, and a model that thinks before it writes will do that. - If the wait is classed
upstreamorlocal-link, look at the network evidence for that moment. One caveat from my own testing: I’ve seen a thinking wait classed asupstream, so check it before you believe it. - If TTFB itself is slow, check round trip and setup. A normal round trip with a slow first byte points away from your network.
- A
–means there’s no TTFT, for example on multiplexed QUIC. watson leaves it blank when it can’t tell.
To list your responses with a first-token wait:
jq -c 'select(.timing.first_token_wait_ms) | {time, app,
ttfb: .timing.ttfb_ms, ttft: .timing.first_token_ms,
wait: .timing.first_token_wait_ms, class: .timing.first_token_wait_class}' \
~/.watson/records.jsonl
Metrics in focus
From my log, October 2 to November 7, for Claude Code answers that streamed for at least a second:
| p50 | p95 | |
|---|---|---|
| TTFB | 952 ms | 3.89 s |
| TTFT | 1.01 s | 4.02 s |
That’s 6,851 streamed answers. In about 90% of them, TTFT is within 5 ms of TTFB, which fits what I saw in the manual check: Claude Code usually sends headers only once the first token is ready. Only 247 had a measurable wait between the preamble and the first token, so the gap shows up rarely in Claude Code. When it does, as in the row at the top, it’s large.
The metrics I use here:
first_token_ms: the inferred first token, from the start of setup on a new connection, otherwise from the request.first_token_wait_msand its class: the silence between preamble and first token.- TTFT first, median and slowest, per turn.
- TTFT p50 and p95, per app, in the exit summary.
And what the method can’t do:
- Thinking shorter than 750 ms reads as a stream that was already flowing, so TTFT comes out early by that much. I saw that once in 146 questions, 399 ms early.
- When the rule falls back to the first byte, the result is a lower bound.
- Time the client spends before it sends the request isn’t on the wire, so it isn’t included.
- Rule 1 assumes TLS 1.3 with a 16 B AEAD tag. TLS 1.2, record padding, or more than 28 WINDOW_UPDATEs in a segment would break it.
- The harness was a Go client. Large contexts on Claude Code’s own client are still untested, since most requests in my check uploaded only about 3.3 KB.
Summary
TTFT in watson is an inference from timing and sizes: 59 of 60 held-out questions landed within ±150 ms of the true first token. It’s always labelled Inferred, never Exact, even in the browser, and it never goes into a score, because the model’s own thinking shouldn’t count against the network.
If I’d read SSE event names in the browser extension, this would have been easy. That was the fallback if calibration failed, and I’m happy I didn’t need it, since everything in watson’s privacy story depends on not reading the stream.