6 months ago
We are experiencing recurring high latency issues on our Singapore deployment, and this is having a direct impact on our business operations and customer experience.
We observed a similar incident yesterday at approximately 23:17 UTC. The issue appeared to improve for a short period afterward, which made us think an infrastructure-side problem may have been addressed. However, the latency has now returned, and requests are again taking significantly longer than normal to complete.
At the moment:
Response times are much higher than expected.
User-facing requests are noticeably delayed.
Business operations are being affected due to the degraded performance.
Could you please investigate whether there are any ongoing issues with the Singapore region, networking, routing, or underlying infrastructure that could be causing this behavior?
Any updates or guidance would be greatly appreciated, as this issue is impacting production traffic.
Railway domain
Khizertikkahall1.railway.up
Khizertikkadc.railway.up
3 Replies
6 months ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • 6 months ago
6 months ago
a month ago
Hi,
I reviewed the reported intermittent latency affecting your Singapore deployment and the attached response-time graph.
Findings
The graph shows repeated p99 latency spikes exceeding 20 seconds, while p50 remains relatively low for much of the observation period. This suggests intermittent tail-latency degradation rather than consistently slow application responses.
The screenshot alone does not establish whether the issue originates in Railway's infrastructure, network routing, application processing, or downstream dependencies.
Recommended investigation
- Open Railway's HTTP logs and filter with
@totalDuration:>1000. - Compare
totalDurationandupstreamRqDurationon the affected requests. If the former is high while the latter remains low, investigate network/proxy overhead. If both rise together, investigate application execution and downstream dependencies. - Send requests with
X-Railway-Debug: 1and captureX-Railway-Edge,X-Railway-Upstream-Zone, andX-Railway-Request-Id. - Verify whether requests are served from
railway/asia-southeast1-eqsg3a, the documented Singapore deployment zone. - Compare the same lightweight endpoint from Singapore and another geographic location, recording DNS, TCP, TLS, time to first byte, total latency, and HTTP status.
- Correlate the high-latency periods with CPU, memory, database latency, connection-pool saturation, and deployment events.
How to isolate the problem
- High proxy duration with low application duration suggests investigating routing or Railway's edge.
- High application duration suggests investigating the application, database, or external services.
- Latency spikes that correlate with a specific edge POP or origin zone provide stronger evidence of a location-specific networking problem.
- Similar application latency across regions suggests looking for a shared dependency or application-level bottleneck.
Next step
Please share a few slow-request HTTP log entries (with sensitive details redacted), including the timing fields and Railway request IDs, alongside the UTC timestamps.
These details would make it possible to narrow down the root cause and provide Railway support with specific evidence.
References: