Skip to content
How would you test a multi-provider AI router in a Railway PR environment?
juniordias00
OP

14 hours ago

At Avetrix Labs, I’m building Neural Void, an app that sends requests to different AI providers. Here’s a change I’d like to evaluate before shipping: suppose we shorten the time the router waits for provider A before trying B. More users might get a fast answer, but we could quietly send more traffic to a pricier model. A green healthcheck wouldn’t tell me whether that trade-off was worth it.

Could we build a small, reusable Railway example to help teams make that decision? My starting point would be a PR environment running the candidate router, with a test runner that sends the same synthetic requests through both the current and proposed routing policies. Two mock providers would follow a fixed script: A is normally fast, sometimes slow, and occasionally returns a rate limit; B answers reliably at a higher simulated cost. No production prompts or provider credentials would be needed to reproduce the comparison.

The case I’d test first: A slows down for 30 seconds, enough requests move to B that B also starts rate-limiting, then A recovers. For each policy I’d compare completed requests, time to first token at the 95th percentile, provider attempts per completed request, and simulated cost per completed request. I’d also check how long it takes traffic to return to A.

If the candidate improves latency for successful replies but completes fewer requests during that burst, would you ship it? What result would make you block the PR, and what would you still want to verify with a small, budget-capped live test?

I’d love to build the example around cases other teams have actually seen. A missed failure mode, a short provider timing trace with synthetic data, or a simpler Railway service layout would give the next person something concrete to test.

$40 Bounty

2 Replies

Railway
BOT

14 hours ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway • about 14 hours ago


safjar

11 hours ago

No, I wouldn't ship it as-is. I ran your scenario through a quick simulation, and the result depended almost entirely on two details that are easy to get wrong in the test setup.

Setup: 5 req/s, A's time to first token jumps to 6–8 s for 30 s, B costs 5× and is capped at 120 req/min, 10 seeds. Current policy waits 8 s for A; the candidate waits 2 s, moves to B, and goes back to A if B rate-limits.

During the slow window:

  • Current (8 s): 100% completed, p95 7.9 s, 1.00 attempts per completed, cost 1.0×
  • Candidate, B limit per minute: 100%, p95 3.1 s, 2.00 attempts, cost 5–6×
  • Candidate, B limit per second, 9 s client deadline: 73%, p95 8.8 s, 3.55 attempts, cost 3.2–5×
  • Candidate + 10% failover budget: 99%, p95 7.9 s, 1.15 attempts, cost 1.3–1.5×

1. How B enforces its limit. If B lets you spend a full minute's quota at once, it absorbs the whole burst and the candidate looks like a clean win: faster, just pricier. Real providers usually don't work that way. Anthropic's docs say a 60 RPM limit "might be enforced as 1 request per second," plus a separate ramp limit when traffic jumps suddenly, which is exactly what happens when everything moves to B. Model that and the picture flips: B rejects, requests go back to slow A, and the candidate ends up slower and completes fewer. A mock B with a simple per-minute counter will tell you to ship something that falls over in production.

2. The client deadline. The drop only appears when "2 s wait + rejected by B + slow A" runs past the deadline. At 10 s everything completed; at 9 s about a quarter failed. Make the deadline part of the scenario, and measure p95 over every request sent, counting failures as the deadline. Measuring only successful replies flatters the candidate.

Would I ship it? If it's faster on successful replies but completes fewer during the burst, I'd block it. Block if, across seeds:

  • burst completion drops more than ~1 percentage point;
  • attempts per completed go above ~1.3 (the router is multiplying its own load);
  • B starts rejecting earlier or for longer than under the current policy;
  • normal-traffic cost goes over budget.

What worked was keeping the short timeout but capping how much traffic can fail over (Google's SRE book uses a 10% retry ratio per client; Envoy defaults to 20%). Once the cap is hit, requests keep waiting on A instead of piling onto B. Completion went back to 99% and cost to ~1.3–1.5×, but most of the latency gain during the slowdown went away too. That's the real trade-off. Respecting Retry-After from B helps as well.

Returning to A: if the router tries A first on every request, it recovers the moment A does. It only matters once you add a circuit breaker; then probe A with a little traffic and ramp back gradually so you don't knock it over again.

Failure modes worth adding:

  • Billing for abandoned requests. Whether A still charges after the router gives up varies by provider and isn't clearly documented. It moved cost from 3.2× to 5× in my run, so simulate both and check it live.
  • Fast first token, then a stall. You need a between-token timeout too, and you can't switch providers once tokens reached the user.
  • Different 429s. Rate limit, ramp limit, and spend cap differ; some lack Retry-After, and a spend cap won't clear on retry.
  • Token limits. Long prompts can hit tokens-per-minute before the request limit.

Railway layout, three services in the PR environment:

  1. Router built from the PR, with both policies inside and a header choosing one per request, so both run on the same build under the same noise.
  2. One mock provider serving A and B from a scenario file and seed, with separate rate-limit counters per policy so both can run at once.
  3. Runner that sends the same schedule through both policies and writes a pass/fail report. Set restart policy to Never.

They talk over private networking (mock-provider.railway.internal); each environment has its own network, available at runtime only, not during build. With Focused PR Environments, untouched mocks may be skipped, so deploy them explicitly. Drop a warm-up window and run several seeds.

The budget-capped live test should only check what mocks can't: real billing for abandoned requests, real rate-limit behavior, and real token timings to tune the mock. Run it on 1–5% of traffic with a hard spend cap at the provider and the router, and stop automatically if a block criterion trips.

Disclosure for my answer above: I used AI to help draft and check it, and reviewed it before posting.


safjar

No, I wouldn't ship it as-is. I ran your scenario through a quick simulation, and the result depended almost entirely on two details that are easy to get wrong in the test setup. Setup: 5 req/s, A's time to first token jumps to 6–8 s for 30 s, B costs 5× and is capped at 120 req/min, 10 seeds. Current policy waits 8 s for A; the candidate waits 2 s, moves to B, and goes back to A if B rate-limits. During the slow window: - **Current (8 s):** 100% completed, p95 7.9 s, 1.00 attempts per completed, cost 1.0× - **Candidate, B limit per minute:** 100%, p95 3.1 s, 2.00 attempts, cost 5–6× - **Candidate, B limit per second, 9 s client deadline:** 73%, p95 8.8 s, 3.55 attempts, cost 3.2–5× - **Candidate + 10% failover budget:** 99%, p95 7.9 s, 1.15 attempts, cost 1.3–1.5× **1. How B enforces its limit.** If B lets you spend a full minute's quota at once, it absorbs the whole burst and the candidate looks like a clean win: faster, just pricier. Real providers usually don't work that way. Anthropic's docs say a 60 RPM limit "might be enforced as 1 request per second," plus a separate ramp limit when traffic jumps suddenly, which is exactly what happens when everything moves to B. Model that and the picture flips: B rejects, requests go back to slow A, and the candidate ends up slower and completes fewer. A mock B with a simple per-minute counter will tell you to ship something that falls over in production. **2. The client deadline.** The drop only appears when "2 s wait + rejected by B + slow A" runs past the deadline. At 10 s everything completed; at 9 s about a quarter failed. Make the deadline part of the scenario, and measure p95 over every request sent, counting failures as the deadline. Measuring only successful replies flatters the candidate. **Would I ship it?** If it's faster on successful replies but completes fewer during the burst, I'd block it. Block if, across seeds: - burst completion drops more than ~1 percentage point; - attempts per completed go above ~1.3 (the router is multiplying its own load); - B starts rejecting earlier or for longer than under the current policy; - normal-traffic cost goes over budget. What worked was keeping the short timeout but capping how much traffic can fail over (Google's SRE book uses a 10% retry ratio per client; Envoy defaults to 20%). Once the cap is hit, requests keep waiting on A instead of piling onto B. Completion went back to 99% and cost to ~1.3–1.5×, but most of the latency gain during the slowdown went away too. That's the real trade-off. Respecting `Retry-After` from B helps as well. **Returning to A:** if the router tries A first on every request, it recovers the moment A does. It only matters once you add a circuit breaker; then probe A with a little traffic and ramp back gradually so you don't knock it over again. **Failure modes worth adding:** - **Billing for abandoned requests.** Whether A still charges after the router gives up varies by provider and isn't clearly documented. It moved cost from 3.2× to 5× in my run, so simulate both and check it live. - **Fast first token, then a stall.** You need a between-token timeout too, and you can't switch providers once tokens reached the user. - **Different 429s.** Rate limit, ramp limit, and spend cap differ; some lack `Retry-After`, and a spend cap won't clear on retry. - **Token limits.** Long prompts can hit tokens-per-minute before the request limit. **Railway layout, three services in the PR environment:** 1. **Router** built from the PR, with both policies inside and a header choosing one per request, so both run on the same build under the same noise. 2. **One mock provider** serving A and B from a scenario file and seed, with separate rate-limit counters per policy so both can run at once. 3. **Runner** that sends the same schedule through both policies and writes a pass/fail report. Set restart policy to Never. They talk over private networking (`mock-provider.railway.internal`); each environment has its own network, available at runtime only, not during build. With Focused PR Environments, untouched mocks may be skipped, so deploy them explicitly. Drop a warm-up window and run several seeds. **The budget-capped live test** should only check what mocks can't: real billing for abandoned requests, real rate-limit behavior, and real token timings to tune the mock. Run it on 1–5% of traffic with a hard spend cap at the provider and the router, and stop automatically if a block criterion trips. Disclosure for my answer above: I used AI to help draft and check it, and reviewed it before posting.

juniordias00
OP

10 hours ago

Thanks, safjar. This is a really useful answer. You’ve taken the scenario beyond “shorter timeout means faster responses” and shown why the deadline, B’s admission limits, and the extra attempts have to be considered together. The failover budget is also a practical direction for the example.

There are a few details I’d like to clarify so we can turn your results into a reproducible test.

First, how was the reported p95 calculated? In the nine-second deadline case, you report 73% completion and a p95 of 8.8 seconds. Later, you recommend assigning the deadline to failed requests. If those figures describe the same request population, assigning nine seconds to the remaining 27% would make its p95 nine seconds. Was the table using successful replies only, a different observation window, or an aggregation across seeds? I’d keep the completion rate and successful-request TTFT visible alongside any metric that assigns a penalty to failures, with each clearly labelled.

Second, when the two-second wait expires, is the original A request cancelled, or does it remain in flight while B is attempted? If B rejects, does “go back to A” mean starting a fresh request or continuing to await the original one? That distinction seems central to both the deadline result and the attempt count. For the budgeted policy, I’d also like to understand whether the router checks the failover budget before abandoning A, so requests that cannot use B can keep waiting.

Could you share the small simulation script or scenario configuration, including B’s limiter and its initial state? At five requests per second, the 30-second slowdown generates roughly 150 arrivals. With a fixed 120-request minute quota, the result can depend on where the burst falls relative to the reset boundary; a bucket that refills continuously behaves differently again. A trace would help us reproduce which requests B accepts and which return to A.

For the 10% failover budget, what denominator and time window did you use? The Google and Envoy references support limiting amplification, although their accounting differs: Google describes a per-client retry ratio, while Envoy’s default budget percentage applies to concurrent retries relative to active and pending requests. Making our example’s accounting explicit would help others reproduce the improvement.

You’ve already identified abandoned-request billing as something to simulate both ways and verify live, which makes sense. The remaining detail I’d want in the report is whether cost and attempts per completion include all work spent on requests that ultimately fail. Results per seed would also help us judge whether the proposed one-percentage-point completion threshold is stable enough to use as a PR gate.

Even a minimal script and one representative request trace would be a valuable next step. Thanks for putting this much thought into it—the answer gives us a concrete foundation for the reusable Railway example.


Welcome!

Sign in to your Railway account to join the conversation.

Loading...