Skip to content
High Latency on Singapore Region Impacting Business Operations
chaudharyali14
OP

6 months ago

We are experiencing recurring high latency issues on our Singapore deployment, and this is having a direct impact on our business operations and customer experience.

We observed a similar incident yesterday at approximately 23:17 UTC. The issue appeared to improve for a short period afterward, which made us think an infrastructure-side problem may have been addressed. However, the latency has now returned, and requests are again taking significantly longer than normal to complete.

At the moment:

Response times are much higher than expected.

User-facing requests are noticeably delayed.

Business operations are being affected due to the degraded performance.

Could you please investigate whether there are any ongoing issues with the Singapore region, networking, routing, or underlying infrastructure that could be causing this behavior?

Any updates or guidance would be greatly appreciated, as this issue is impacting production traffic.

Railway domain

Khizertikkahall1.railway.up

Khizertikkadc.railway.up

$20 Bounty

3 Replies

Railway
BOT

6 months ago

This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.

Status changed to Open Railway • 6 months ago


chaudharyali14
OP

6 months ago

2 - Copy.png


bhaktiaifiles

a month ago

Try this app, it could help https://launchfield.app/dashboard


nilu-mukherjee

a month ago

Hi,

I reviewed the reported intermittent latency affecting your Singapore deployment and the attached response-time graph.

Findings

The graph shows repeated p99 latency spikes exceeding 20 seconds, while p50 remains relatively low for much of the observation period. This suggests intermittent tail-latency degradation rather than consistently slow application responses.

The screenshot alone does not establish whether the issue originates in Railway's infrastructure, network routing, application processing, or downstream dependencies.

Recommended investigation

  1. Open Railway's HTTP logs and filter with @totalDuration:>1000.
  2. Compare totalDuration and upstreamRqDuration on the affected requests. If the former is high while the latter remains low, investigate network/proxy overhead. If both rise together, investigate application execution and downstream dependencies.
  3. Send requests with X-Railway-Debug: 1 and capture X-Railway-Edge, X-Railway-Upstream-Zone, and X-Railway-Request-Id.
  4. Verify whether requests are served from railway/asia-southeast1-eqsg3a, the documented Singapore deployment zone.
  5. Compare the same lightweight endpoint from Singapore and another geographic location, recording DNS, TCP, TLS, time to first byte, total latency, and HTTP status.
  6. Correlate the high-latency periods with CPU, memory, database latency, connection-pool saturation, and deployment events.

How to isolate the problem

  • High proxy duration with low application duration suggests investigating routing or Railway's edge.
  • High application duration suggests investigating the application, database, or external services.
  • Latency spikes that correlate with a specific edge POP or origin zone provide stronger evidence of a location-specific networking problem.
  • Similar application latency across regions suggests looking for a shared dependency or application-level bottleneck.

Next step

Please share a few slow-request HTTP log entries (with sensitive details redacted), including the timing fields and Railway request IDs, alongside the UTC timestamps.

These details would make it possible to narrow down the root cause and provide Railway support with specific evidence.

References:


Welcome!

Sign in to your Railway account to join the conversation.

Loading...