5 months ago
Hi Railway team,
We're seeing intermittent latency spikes on our production service running in the Singapore region.
Symptoms
Most requests complete in around 95–100ms, but approximately 25% of requests suddenly take 2–8 seconds.
This occurs even on a lightweight health endpoint:
GET /api/healthThere is no database query or heavy processing involved.
Reproduction
Running the following command repeatedly from Japan:
for i in $(seq 100); do
curl -sS -o /dev/null \
-w "%{time_total}\n" \
--resolve buy-buy.pro:443:<SERVICE_IP> \
https://buy-buy.pro/api/health
doneTypical output:
0.097
0.101
0.096
5.616
0.098
4.525
0.097
1.341
0.099
2.108
...Observed distribution:
- Median: ~99ms
- About 25% of requests exceed 1 second
- Many requests take 4–8 seconds
- Maximum observed: ~8 seconds
The issue is reproducible across multiple runs.
Environment
- Region: Singapore
- Client location: Japan
- HTTPS
- Production service
- Using
--resolveto bypass DNS, so DNS lookup is not part of the latency.
We've also noticed recent Singapore networking incidents on the Railway status page, so we're wondering if this could be related to the current edge or transit infrastructure.
Could you please investigate whether there are any known networking or routing issues affecting the Singapore region?
If needed, we're happy to provide:
- Project ID
- Service ID
- Exact timestamps (UTC)
- Public IP used for testing
- Additional latency logs
Thank you!
3 Replies
Status changed to Awaiting Railway Response Railway • 5 months ago
5 months ago
Additional information after further investigation:
We performed an isolation test.
- The PostgreSQL database is still running on Railway in the Singapore region.
- We migrated only the Next.js web server to another hosting provider in Singapore.
- The application still connects to the same Railway PostgreSQL instance.
- After this change, the intermittent 2–8 second latency spikes completely disappeared.
This suggests the issue is unlikely to be caused by:
- The database itself
- Connectivity between Japan and Singapore in general
Instead, it appears to be related to the Railway-hosted application service, its networking, routing, or edge infrastructure.
Hopefully this helps narrow down the investigation.
Status changed to Awaiting Railway Response Railway • 5 months ago
5 months ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • 5 months ago
a month ago
Hi there!
That isolation test is exceptional work—migrating the web server off Railway while keeping the database on Railway and having the 2–8 second latency completely disappear definitively rules out database locking, query performance, and cross-border network transit between Japan and Singapore.
Since a stateless health endpoint (/api/health) with zero database queries or heavy processing still experiences a 25% tail latency spike (4–8 seconds), the bottleneck is isolated entirely within the Railway-hosted application environment or its regional ingress layer in Singapore.
Here is how to definitively isolate and resolve the root cause:
- Deploy a Zero-Dependency Baseline
App vs Platform isolation
Deploy a barebones, single-file static HTTP server (e.g., a basic Python http.server or Go net/http binary returning a static 200 OK) to the exact same Railway service in Singapore. Run your 100-request curl test against it.
- Verification: If the minimal server responds in <15ms consistently, your Next.js application runtime (middleware, background async tasks, or event loop blocking) is causing the stalls. If the 2–8 second spikes persist even on the barebones server, the bottleneck is platform-side (container cgroup CPU throttling or regional edge proxy routing).
- Inspect Container Metrics and Resource Limits
Railway Dashboard
Go to your Railway project dashboard, open the Metrics tab for the Singapore service, and review CPU and memory utilization during the test run.
- Verification: Check if CPU usage hits 100% or if cgroup throttling events correlate precisely with the timestamps of your 2–8 second latency values. If constrained, scale up the service's resource tier.
- Provide Diagnostic Telemetry to Railway Support
Escalation
If metrics look healthy and a barebones server still exhibits spikes, provide your Service ID, precise UTC timestamps of the curl test runs, and your testing IP to Railway support so they can inspect regional ingress routing logs on their Singapore edge nodes.
- Verification: Support can trace whether edge load balancers are queuing incoming requests before hitting your container instance.
Official References
- Railway Documentation: Metrics & Observability
- Railway Documentation: Troubleshooting Deployments
Disclosure: I used AI assistance to help prepare this response and reviewed the technical reasoning before submitting.
douglasluyinda
Hi there! That isolation test is exceptional work—migrating the web server off Railway while keeping the database on Railway and having the 2–8 second latency completely disappear definitively rules out database locking, query performance, and cross-border network transit between Japan and Singapore. Since a stateless health endpoint (/api/health) with zero database queries or heavy processing still experiences a 25% tail latency spike (4–8 seconds), the bottleneck is isolated entirely within the Railway-hosted application environment or its regional ingress layer in Singapore. Here is how to definitively isolate and resolve the root cause: 1. Deploy a Zero-Dependency Baseline App vs Platform isolation Deploy a barebones, single-file static HTTP server (e.g., a basic Python http.server or Go net/http binary returning a static 200 OK) to the exact same Railway service in Singapore. Run your 100-request curl test against it. * Verification: If the minimal server responds in <15ms consistently, your Next.js application runtime (middleware, background async tasks, or event loop blocking) is causing the stalls. If the 2–8 second spikes persist even on the barebones server, the bottleneck is platform-side (container cgroup CPU throttling or regional edge proxy routing). 2. Inspect Container Metrics and Resource Limits Railway Dashboard Go to your Railway project dashboard, open the Metrics tab for the Singapore service, and review CPU and memory utilization during the test run. * Verification: Check if CPU usage hits 100% or if cgroup throttling events correlate precisely with the timestamps of your 2–8 second latency values. If constrained, scale up the service's resource tier. 3. Provide Diagnostic Telemetry to Railway Support Escalation If metrics look healthy and a barebones server still exhibits spikes, provide your Service ID, precise UTC timestamps of the curl test runs, and your testing IP to Railway support so they can inspect regional ingress routing logs on their Singapore edge nodes. * Verification: Support can trace whether edge load balancers are queuing incoming requests before hitting your container instance. Official References * Railway Documentation: Metrics & Observability * Railway Documentation: Troubleshooting Deployments Disclosure: I used AI assistance to help prepare this response and reviewed the technical reasoning before submitting.
a month ago
The migration is a strong clue, but it doesn’t isolate the slow part of Railway’s request path by itself. I’d correlate a few slow requests with Railway’s logs: compare totalDuration with upstreamRqDuration. High total time with low upstream time points to the network or proxy path; high upstream time points to the app.
Also, --resolve skips DNS only. Each separate curl still opens a new TCP/TLS connection, and time_total combines connection, TLS, and response time. Could you capture time_connect, time_appconnect, and time_starttransfer, plus X-Railway-Edge and X-Railway-Upstream-Zone, for both fast and slow requests? Those values should help Railway trace where the delay occurs.
Railway’s slow request guide explains the log timings; its edge networking guide explains the edge headers.