We're seeing intermittent latency spikes on our production service running in the Singapore region.
unit-blue
PROOP

5 days ago

Hi Railway team,

We're seeing intermittent latency spikes on our production service running in the Singapore region.

Symptoms

Most requests complete in around 95–100ms, but approximately 25% of requests suddenly take 2–8 seconds.

This occurs even on a lightweight health endpoint:

GET /api/health

There is no database query or heavy processing involved.

Reproduction

Running the following command repeatedly from Japan:

for i in $(seq 100); do
  curl -sS -o /dev/null \
    -w "%{time_total}\n" \
    --resolve buy-buy.pro:443:<SERVICE_IP> \
    https://buy-buy.pro/api/health
done

Typical output:

0.097
0.101
0.096
5.616
0.098
4.525
0.097
1.341
0.099
2.108
...

Observed distribution:

  • Median: ~99ms
  • About 25% of requests exceed 1 second
  • Many requests take 4–8 seconds
  • Maximum observed: ~8 seconds

The issue is reproducible across multiple runs.

Environment

  • Region: Singapore
  • Client location: Japan
  • HTTPS
  • Production service
  • Using --resolve to bypass DNS, so DNS lookup is not part of the latency.

We've also noticed recent Singapore networking incidents on the Railway status page, so we're wondering if this could be related to the current edge or transit infrastructure.

Could you please investigate whether there are any known networking or routing issues affecting the Singapore region?

If needed, we're happy to provide:

  • Project ID
  • Service ID
  • Exact timestamps (UTC)
  • Public IP used for testing
  • Additional latency logs

Thank you!

$20 Bounty

1 Replies

Status changed to Awaiting Railway Response Railway 5 days ago


unit-blue
PROOP

4 days ago

Additional information after further investigation:

We performed an isolation test.

  • The PostgreSQL database is still running on Railway in the Singapore region.
  • We migrated only the Next.js web server to another hosting provider in Singapore.
  • The application still connects to the same Railway PostgreSQL instance.
  • After this change, the intermittent 2–8 second latency spikes completely disappeared.

This suggests the issue is unlikely to be caused by:

  • The database itself
  • Connectivity between Japan and Singapore in general

Instead, it appears to be related to the Railway-hosted application service, its networking, routing, or edge infrastructure.

Hopefully this helps narrow down the investigation.


Status changed to Awaiting Railway Response Railway 4 days ago


Railway
BOT

3 days ago

This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.

Status changed to Open Railway 3 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...