Skip to content
We're seeing intermittent latency spikes on our production service running in the Singapore region.
unit-blue
OP

5 months ago

Hi Railway team,

We're seeing intermittent latency spikes on our production service running in the Singapore region.

Symptoms

Most requests complete in around 95–100ms, but approximately 25% of requests suddenly take 2–8 seconds.

This occurs even on a lightweight health endpoint:

GET /api/health

There is no database query or heavy processing involved.

Reproduction

Running the following command repeatedly from Japan:

for i in $(seq 100); do
  curl -sS -o /dev/null \
    -w "%{time_total}\n" \
    --resolve buy-buy.pro:443:<SERVICE_IP> \
    https://buy-buy.pro/api/health
done

Typical output:

0.097
0.101
0.096
5.616
0.098
4.525
0.097
1.341
0.099
2.108
...

Observed distribution:

  • Median: ~99ms
  • About 25% of requests exceed 1 second
  • Many requests take 4–8 seconds
  • Maximum observed: ~8 seconds

The issue is reproducible across multiple runs.

Environment

  • Region: Singapore
  • Client location: Japan
  • HTTPS
  • Production service
  • Using --resolve to bypass DNS, so DNS lookup is not part of the latency.

We've also noticed recent Singapore networking incidents on the Railway status page, so we're wondering if this could be related to the current edge or transit infrastructure.

Could you please investigate whether there are any known networking or routing issues affecting the Singapore region?

If needed, we're happy to provide:

  • Project ID
  • Service ID
  • Exact timestamps (UTC)
  • Public IP used for testing
  • Additional latency logs

Thank you!

$20 Bounty

3 Replies

Status changed to Awaiting Railway Response Railway • 5 months ago


unit-blue
OP

5 months ago

Additional information after further investigation:

We performed an isolation test.

  • The PostgreSQL database is still running on Railway in the Singapore region.
  • We migrated only the Next.js web server to another hosting provider in Singapore.
  • The application still connects to the same Railway PostgreSQL instance.
  • After this change, the intermittent 2–8 second latency spikes completely disappeared.

This suggests the issue is unlikely to be caused by:

  • The database itself
  • Connectivity between Japan and Singapore in general

Instead, it appears to be related to the Railway-hosted application service, its networking, routing, or edge infrastructure.

Hopefully this helps narrow down the investigation.


Status changed to Awaiting Railway Response Railway • 5 months ago


Railway
BOT

5 months ago

This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.

Status changed to Open Railway • 5 months ago


douglasluyinda

a month ago

Hi there!

That isolation test is exceptional work—migrating the web server off Railway while keeping the database on Railway and having the 2–8 second latency completely disappear definitively rules out database locking, query performance, and cross-border network transit between Japan and Singapore.

Since a stateless health endpoint (/api/health) with zero database queries or heavy processing still experiences a 25% tail latency spike (4–8 seconds), the bottleneck is isolated entirely within the Railway-hosted application environment or its regional ingress layer in Singapore.

Here is how to definitively isolate and resolve the root cause:

  1. Deploy a Zero-Dependency Baseline

App vs Platform isolation

Deploy a barebones, single-file static HTTP server (e.g., a basic Python http.server or Go net/http binary returning a static 200 OK) to the exact same Railway service in Singapore. Run your 100-request curl test against it.

  • Verification: If the minimal server responds in <15ms consistently, your Next.js application runtime (middleware, background async tasks, or event loop blocking) is causing the stalls. If the 2–8 second spikes persist even on the barebones server, the bottleneck is platform-side (container cgroup CPU throttling or regional edge proxy routing).
  1. Inspect Container Metrics and Resource Limits

Railway Dashboard

Go to your Railway project dashboard, open the Metrics tab for the Singapore service, and review CPU and memory utilization during the test run.

  • Verification: Check if CPU usage hits 100% or if cgroup throttling events correlate precisely with the timestamps of your 2–8 second latency values. If constrained, scale up the service's resource tier.
  1. Provide Diagnostic Telemetry to Railway Support

Escalation

If metrics look healthy and a barebones server still exhibits spikes, provide your Service ID, precise UTC timestamps of the curl test runs, and your testing IP to Railway support so they can inspect regional ingress routing logs on their Singapore edge nodes.

  • Verification: Support can trace whether edge load balancers are queuing incoming requests before hitting your container instance.

Official References

  • Railway Documentation: Metrics & Observability
  • Railway Documentation: Troubleshooting Deployments

Disclosure: I used AI assistance to help prepare this response and reviewed the technical reasoning before submitting.


douglasluyinda

Hi there! That isolation test is exceptional work—migrating the web server off Railway while keeping the database on Railway and having the 2–8 second latency completely disappear definitively rules out database locking, query performance, and cross-border network transit between Japan and Singapore. Since a stateless health endpoint (/api/health) with zero database queries or heavy processing still experiences a 25% tail latency spike (4–8 seconds), the bottleneck is isolated entirely within the Railway-hosted application environment or its regional ingress layer in Singapore. Here is how to definitively isolate and resolve the root cause: 1. Deploy a Zero-Dependency Baseline App vs Platform isolation Deploy a barebones, single-file static HTTP server (e.g., a basic Python http.server or Go net/http binary returning a static 200 OK) to the exact same Railway service in Singapore. Run your 100-request curl test against it. * Verification: If the minimal server responds in <15ms consistently, your Next.js application runtime (middleware, background async tasks, or event loop blocking) is causing the stalls. If the 2–8 second spikes persist even on the barebones server, the bottleneck is platform-side (container cgroup CPU throttling or regional edge proxy routing). 2. Inspect Container Metrics and Resource Limits Railway Dashboard Go to your Railway project dashboard, open the Metrics tab for the Singapore service, and review CPU and memory utilization during the test run. * Verification: Check if CPU usage hits 100% or if cgroup throttling events correlate precisely with the timestamps of your 2–8 second latency values. If constrained, scale up the service's resource tier. 3. Provide Diagnostic Telemetry to Railway Support Escalation If metrics look healthy and a barebones server still exhibits spikes, provide your Service ID, precise UTC timestamps of the curl test runs, and your testing IP to Railway support so they can inspect regional ingress routing logs on their Singapore edge nodes. * Verification: Support can trace whether edge load balancers are queuing incoming requests before hitting your container instance. Official References * Railway Documentation: Metrics & Observability * Railway Documentation: Troubleshooting Deployments Disclosure: I used AI assistance to help prepare this response and reviewed the technical reasoning before submitting.

iiviavs

a month ago

The migration is a strong clue, but it doesn’t isolate the slow part of Railway’s request path by itself. I’d correlate a few slow requests with Railway’s logs: compare totalDuration with upstreamRqDuration. High total time with low upstream time points to the network or proxy path; high upstream time points to the app.

Also, --resolve skips DNS only. Each separate curl still opens a new TCP/TLS connection, and time_total combines connection, TLS, and response time. Could you capture time_connect, time_appconnect, and time_starttransfer, plus X-Railway-Edge and X-Railway-Upstream-Zone, for both fast and slow requests? Those values should help Railway trace where the delay occurs.

Railway’s slow request guide explains the log timings; its edge networking guide explains the edge headers.


Welcome!

Sign in to your Railway account to join the conversation.

Loading...