A cascade beats running both models
Inference is paid forever, training once. A router that only escalates when it has to is several times cheaper, at a cost that shows up in the tail rather than the average.
The expected cost of a cascade is the small model plus the router plus the probability of escalation times the large model. With a quarter of requests escalating and an eight times more expensive path, that comes to roughly 2.7 times cheaper than running both every time.
The catch is not in the average. A cascade lowers the median and raises the tail, because an escalated request pays for both models in sequence. For an interactive product, a worse p99 can be a net loss even when the mean improves, so both numbers get measured and neither alone is allowed to decide.
The second saving is larger and less discussed. Evidence embeddings are static. Computing them inside the request path is paying repeatedly for an answer that does not change, so they are precomputed into an index and the request path only does a lookup.
Quantisation follows the same logic, with one exception. The backbone runs in low precision, the coupling does not: int8 in the coupling destroys the round trip and with it the invertibility the whole construction depends on.