a month ago
Project: motivated-victory (ID: f5dd1eff-a053-4c46-834e-e5e22ba8c381)
Service: web (staging environment)
Plan: Pro
Summary: Our FastAPI/uvicorn service (4 workers) shows severe request queuing under moderate concurrent load (as low as 50-200 simultaneous requests), even though the container's own CPU usage stays near-idle (~0.1 vCPU) throughout. Requests eventually succeed (200 OK) — this is not an application error — but latency climbs from ~4.7s to 15s+ as concurrency increases, which is unacceptable for our production launch.
Evidence gathered:
50 truly-parallel curl requests to the same endpoint: 100% returned 200, but response times climbed from 4.7s (first) to 15.0s (last) — a clear queuing pattern, not errors
CPU/RAM metrics during this exact load window show negligible usage (~0.1 vCPU peak), ruling out compute saturation as the cause
Individual, isolated curl requests during a separate concurrent-load test (k6, 200+ VUs) returned fast, healthy 200s with real edge headers (x-railway-edge: sin1), even while the concurrent load was reporting client-side timeouts — suggesting requests are being queued somewhere (proxy layer?) rather than processed with genuine parallelism
We've already fixed two blocking synchronous calls in our own app code (confirmed via isolated testing: real ~6-7x latency improvement per-request) and increased uvicorn --workers from 1 to 4 — neither meaningfully changed the queuing behavior under concurrent load
Sample request IDs from healthy individual requests during a failing load window: t2yCe0cjQIutEnEx9I3ezw, 30Tu78fLRDuGkWjBlt7tkg, PevPCjpSRJapLkO6npoFkQ
Question: Is there a platform-level concurrency/connection limit (per-service, per-replica, or per-IP) that would explain requests queuing this severely at only 50-200 concurrent connections, despite the container showing idle CPU? If so, is this configurable, or does it require Enterprise/a different setup to raise?
We have a production launch this week and need to understand whether this is something we can resolve via configuration, or whether it's a platform ceiling requiring a different approach.
6 Replies
a month ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • about 1 month ago
a month ago
Any update ?
a month ago
-
Distribute k6 load test across multiple IPs or run the test within railway private network to bypass public edge proxy entirely
-
Run --workers 4 directly using Uvicorn worker class
gunicorn main:app -w 4 -k uvicorn.workers.UvicornWorker
-
Temporarily crank up ORM connection pool_size and max_overflow in SQLAlchemy.
-
change your endpoint definition, from async def to standard def. FastAPI will then automatically push def endpoints to an external threadpool, leading to prevention on blocking the main async event loop.
h701h
1. Distribute k6 load test across multiple IPs or run the test within railway private network to bypass public edge proxy entirely 2. Run --workers 4 directly using Uvicorn worker class gunicorn main:app -w 4 -k uvicorn.workers.UvicornWorker 3. Temporarily crank up ORM connection pool_size and max_overflow in SQLAlchemy. 4. change your endpoint definition, from async def to standard def. FastAPI will then automatically push def endpoints to an external threadpool, leading to prevention on blocking the main async event loop.
a month ago
Thanks for the suggestions. We tested #1 directly and have a clear, decisive result:
We ran the identical load test (k6, ramping to 2000 VUs, same endpoint, same app code, same 4 workers, same database) two ways:
Via the public URL:
Failure rate: ~79-80%
p95 latency: 10s (timeout ceiling)
Via Railway's private network (deployed a temporary k6-runner service in the same project, targeting web.railway.internal:8080 directly):
Failure rate: 0.89% (714 of 79,620 requests)
p95 latency: 1.85s
p90: 1.0s, avg: 497ms
Throughput: 263.5 req/s
Both k6 thresholds (error rate <5%, p95 <3000ms) passed cleanly
This isolates the bottleneck conclusively: with everything else held constant, only the network path changed, and the difference was ~90x in failure rate and ~5x in latency. The app itself handles this load fine — something in the public ingress/edge/proxy layer cannot.
Given this, could you help us understand: is there a connection or rate limit on the public edge proxy (per-service, per-plan-tier, or per-IP) that would explain this? Is this something configurable on our end, or a platform-level ceiling tied to our current plan? We have a production launch on July 21st and need to resolve this before then.
a month ago
We implemented suggestion #2 (gunicorn + uvicorn_worker.UvicornWorker, matching your recommended config exactly, including correct --keep-alive/--timeout flag mapping) and re-ran the identical load test against the public production URL:
Failure rate: 79.80% (virtually unchanged from before the gunicorn switch)
p95 latency: 10s (unchanged, timeout ceiling)
This is the same result as our original uvicorn --workers 4 setup — the process manager change had no measurable effect on the public-facing number.
Combined with our earlier private-network test (0.89% failure, p95=1.85s, same app code, same load, same everything except the network path), we've now ruled out every app-side variable available to us: worker count, worker process manager, and the async blocking-call issues we fixed earlier. The bottleneck is conclusively isolated to something in the public ingress/edge/proxy path — we've done what we can from the application side.
Given this, we really need your team's help identifying the specific platform-level limit (connection concurrency, rate limiting, edge capacity tied to our plan tier, or something else) so we can either configure around it or understand what upgrade/change would resolve it
h701h
1. Distribute k6 load test across multiple IPs or run the test within railway private network to bypass public edge proxy entirely 2. Run --workers 4 directly using Uvicorn worker class gunicorn main:app -w 4 -k uvicorn.workers.UvicornWorker 3. Temporarily crank up ORM connection pool_size and max_overflow in SQLAlchemy. 4. change your endpoint definition, from async def to standard def. FastAPI will then automatically push def endpoints to an external threadpool, leading to prevention on blocking the main async event loop.
a month ago
Did you try the remaining points? I think 3rd point is the most reasonable thing to do now
a month ago
Also capture edge time headers and record them for every failed request ( X-Railway-Request-Id, X-Request-Start, X-Railway-Edge)