17 days ago
Hi Railway team,
We are investigating repeatable latency differences between two services in asia-southeast1-eqsg3a.
Concurrent diagnostic probes run via SSH on 2026-09-26 at approximately 18:27 UTC showed:
-
Python time.sleep(0.01) wake-up overshoot — 100 samples:
Control: p95 0.140 ms, max 0.517 ms
Candidate: p95 16.918 ms, max 23.708 ms
-
Persistent read-only PostgreSQL SELECT 1 — 100 samples:
Control round-trip: p95 2.570 ms, max 4.718 ms
Candidate round-trip: p95 5.658 ms, max 28.137 ms
-
EXPLAIN ANALYZE SELECT 1 — 50 samples:
Control round-trip p95: 2.272 ms
Candidate round-trip p95: 19.297 ms
PostgreSQL executor p95: 0.028 ms and 0.019 ms respectively.
The sleep probe does not import application code, access the database, or use the network. The DB probes bypass the application connection pool.
Both containers report:
- cpu.max: 800000 100000
- cpu.weight: 100
- Timer slack: 50000 ns
- Normal scheduling policy
- Clocksource: kvm-clock
- Zero CPU quota-throttling increments during these probes
Observed kernel versions:
- Control: 6.18.15+deb13-cloud-amd64
- Candidate: 6.12.22+bpo-cloud-amd64
We are not claiming the kernel difference causes the problem. The application images also have three differing Python dependency versions, so we are not claiming identical runtime builds.
We restarted only the candidate using its exact existing image. Its deployment SHA/image remained unchanged, but the no-network wake-up jitter persisted.
Could Railway investigate scheduling/placement for the candidate, or advise how to move only that candidate to an equivalent healthy runtime in the same Singapore region while preserving its image?
Please leave the control service, shared database, and normal traffic unchanged. This is an isolated candidate diagnostic, not a production outage.
We can provide exact service/deployment identifiers through an appropriate private channel.
Thank you.
1 Replies
Status changed to Awaiting Railway Response Railway • 17 days ago
16 days ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 16 days ago
15 days ago
Your data localizes this pretty well: the 16–24 ms tail is below the application and database layers.
time.sleep(0.01) exercises timer expiry plus scheduler wake-up. Python only guarantees that a sleep lasts at least the requested duration; its documentation explicitly notes that suspension can be longer by an arbitrary amount because of system scheduling. Since this probe imports no application code and uses no network or database, the three dependency differences cannot explain that result.
Likewise, PostgreSQL execution staying around 0.02 ms while client-observed round trips reach 19–28 ms places the delay outside the PostgreSQL executor. Client scheduling delay can inflate that round trip even when the server finishes immediately.
Zero nr_throttled/throttled_usec increments rules out CFS quota throttling during the sample, but it does not rule out run-queue delay, hypervisor steal time, IRQ/softirq pressure, host timer latency, or CPU idle-exit latency. A 50 µs timer slack also cannot account for a repeatable ~17 ms p95.
The kernel difference is relevant evidence even though it does not prove causation: containers share the host kernel. Restarting the exact same image preserves userspace, but it does not make the two runtimes equivalent when one still reports 6.12 and the control reports 6.18. Railway’s docs distinguish restarting an existing deployment from redeploying it, but I cannot find any documented guarantee that either operation forces placement onto a different physical host.
I would capture these concurrently on both services during another bounded run:
/proc/pressure/cpuvmstat 1(r,st,in, andcs)/proc/schedstat/sys/fs/cgroup/cpu.statRAILWAY_REPLICA_ID, deployment ID,uname -a, and exact UTC timestamps
If permitted in the container, cyclictest or the kernel timerlat tracer is the direct test for timer-to-thread wake-up latency. Even without those permissions, correlated CPU PSI, steal time, and scheduler counters would help Railway identify whether the delay is guest scheduling or host-side.
For relocation, Railway publicly documents changing an individual service’s region and changing replica count. I do not see a documented customer control for “evict this replica to another host in the same region,” nor a guarantee that restart/redeploy changes the host. Because you need the same Singapore region and want the control, database, and normal traffic untouched, the clean remedy is for Railway to re-place only this candidate internally while retaining the image, then repeat the no-network probe. I would send the private service/deployment IDs plus the timestamped comparison above; the differing host kernels and persistent no-network wake-up tail are the strongest evidence for that request.
References:
- Python
time.sleep: https://docs.python.org/3/library/time.html#time.sleep - Railway restart vs. redeploy: https://docs.railway.com/cli/restart
- Railway regions: https://docs.railway.com/deployments/regions
- Railway scaling/replicas: https://docs.railway.com/deployments/scaling
- Linux timer-latency tracer: https://docs.kernel.org/trace/timerlat-tracer.html