3 hours ago
Hi Railway team,
We are investigating repeatable latency differences between two services in asia-southeast1-eqsg3a.
Concurrent diagnostic probes run via SSH on 2026-09-26 at approximately 18:27 UTC showed:
-
Python time.sleep(0.01) wake-up overshoot — 100 samples:
Control: p95 0.140 ms, max 0.517 ms
Candidate: p95 16.918 ms, max 23.708 ms
-
Persistent read-only PostgreSQL SELECT 1 — 100 samples:
Control round-trip: p95 2.570 ms, max 4.718 ms
Candidate round-trip: p95 5.658 ms, max 28.137 ms
-
EXPLAIN ANALYZE SELECT 1 — 50 samples:
Control round-trip p95: 2.272 ms
Candidate round-trip p95: 19.297 ms
PostgreSQL executor p95: 0.028 ms and 0.019 ms respectively.
The sleep probe does not import application code, access the database, or use the network. The DB probes bypass the application connection pool.
Both containers report:
- cpu.max: 800000 100000
- cpu.weight: 100
- Timer slack: 50000 ns
- Normal scheduling policy
- Clocksource: kvm-clock
- Zero CPU quota-throttling increments during these probes
Observed kernel versions:
- Control: 6.18.15+deb13-cloud-amd64
- Candidate: 6.12.22+bpo-cloud-amd64
We are not claiming the kernel difference causes the problem. The application images also have three differing Python dependency versions, so we are not claiming identical runtime builds.
We restarted only the candidate using its exact existing image. Its deployment SHA/image remained unchanged, but the no-network wake-up jitter persisted.
Could Railway investigate scheduling/placement for the candidate, or advise how to move only that candidate to an equivalent healthy runtime in the same Singapore region while preserving its image?
Please leave the control service, shared database, and normal traffic unchanged. This is an isolated candidate diagnostic, not a production outage.
We can provide exact service/deployment identifiers through an appropriate private channel.
Thank you.
0 Replies
Status changed to Awaiting Railway Response Railway • about 3 hours ago