13 hours ago
Everything had been working fine until recently, but after a recent deployment of my Next.js app to production (Southeast Asia🇸🇬), I started experiencing a massive drop in performance and extremely slow network communication.
Even though there were very few code changes in this deployment, the build process took significantly longer than usual. Since I haven't made any major changes on my end, I suspect there might be an issue on Railway's side.
Right now, the performance and network connections remain highly unstable and terribly slow, and this issue has persisted continuously since the deployment.
Could you please look into this to see if there are any ongoing issues or anomalies with the service?
Details of the situation:
・I deployed the app around 3:00 AM today (JST), and the performance has been consistently poor ever since.
・The build took noticeably longer than usual, which already seemed abnormal.
・As I have stated multiple times, I have not made any changes to the code or versions.
・There is a clear, drastic drop in overall performance compared to before the deployment.
・Delays are occurring even on simple static pages, and there is massive latency on the API as well.
・Furthermore, regarding the API (App Router), since it only handles very lightweight processes, the response time used to be consistently under 1 second. However, it now sometimes takes more than 10 seconds.
・The API includes database operations, but I have confirmed there are zero delays on the DB side.
Based on all this, it is highly obvious that there is an issue on Railway's end. Please investigate this properly.
As shown in the 1-week log in the attached image.png, there is a sudden drop in memory right after the deployment. Actually, there are two sudden drops in memory in the image (the most recent one being the one I am referring to), and both of these occurred immediately after a deployment. Notably, the older one (about a week ago) was also accompanied by a drastic performance drop immediately after that deployment, just like this time. Could this be related to the issue?
17 Replies
13 hours ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 13 hours ago
13 hours ago
Hey, this was almost certainly on Railway's side there was an incident on their Edge Network causing elevated latency, with partial outages in US West (California) and US East (Virginia):
https://status.railway.com/incident/A97BS3FC
Since it's been resolved for a while now, trigger a fresh redeploy and see if things go back to normal. If it's still slow after that, it's likely unrelated to the incident worth checking your recent changes or opening a ticket with Railway support.
13 hours ago
No, It is the South Asia region.
13 hours ago
It is unrelated to that outage.
13 hours ago
Ah try this:
Go to your service -> Deployments -> find the last deployment that was fast -> Redeploy it. If:
- Fast again -> the problem is in your new build, not Railway
- Still slow -> platform/infra side, open a ticket
13 hours ago
Yes, it remains just as slow. As I mentioned earlier, I made zero changes to my code. The problem is clearly with your service.
13 hours ago
Can i have the public url of your project ?
13 hours ago
FWIW I tested your app from Europe: TTFB is ~240ms after connect, which is healthy for a route through Europe to a Southeast Asia origin. So the origin itself responds fine on this route the slowness is either specific to your network path, or to dynamic/API routes. Can you tell where exactly you see it: page loads in the browser, API responses, or server-side calls to a DB/external service?
13 hours ago
When it's working properly, the speed is normal and fast. What can you possibly tell from just a single test? It's taking an unbelievable amount of time.
Attachments
13 hours ago
It seems it's sometimes taking over 2 seconds to load just a simple static page. This is absolutely ridiculous.
Attachments
12 hours ago
Details of the situation:
・I deployed the app around 3:00 AM today (JST), and the performance has been consistently poor ever since.
・The build took noticeably longer than usual, which already seemed abnormal.
・As I have stated multiple times, I have not made any changes to the code or versions.
・There is a clear, drastic drop in overall performance compared to before the deployment.
・Delays are occurring even on simple static pages, and there is massive latency on the API as well.
・Furthermore, regarding the API (App Router), since it only handles very lightweight processes, the response time used to be consistently under 1 second. However, it now sometimes takes more than 10 seconds.
・The API includes database operations, but I have confirmed there are zero delays on the DB side.
Based on all this, it is highly obvious that there is an issue on Railway's end. Please investigate this properly.
Attachments
12 hours ago
I can't pinpoint exactly where the bottleneck is, but it seems like the delay is happening at the routing level.
12 hours ago
I am seeing similar delays in the past hour-ish, from US
12 hours ago
As shown in the 1-week log in the attached image.png, there is a sudden drop in memory right after the deployment. Actually, there are two sudden drops in memory in the image (the most recent one being the one I am referring to), and both of these occurred immediately after a deployment. Notably, the older one (about a week ago) was also accompanied by a drastic performance drop immediately after that deployment, just like this time. Could this be related to the issue?
Attachments
sheitabrk
FWIW I tested your app from Europe: TTFB is ~240ms after connect, which is healthy for a route through Europe to a Southeast Asia origin. So the origin itself responds fine on this route the slowness is either specific to your network path, or to dynamic/API routes. Can you tell where exactly you see it: page loads in the browser, API responses, or server-side calls to a DB/external service?
11 hours ago
As shown in the 1-week log in the attached image.png, there is a sudden drop in memory right after the deployment. Actually, there are two sudden drops in memory in the image (the most recent one being the one I am referring to), and both of these occurred immediately after a deployment. Notably, the older one (about a week ago) was also accompanied by a drastic performance drop immediately after that deployment, just like this time. Could this be related to the issue?
Attachments
2 hours ago
As shown in the 1-week log in the attached image.png, there is a sudden drop in memory right after the deployment. Actually, there are two sudden drops in memory in the image (the most recent one being the one I am referring to), and both of these occurred immediately after a deployment. Notably, the older one (about a week ago) was also accompanied by a drastic performance drop immediately after that deployment, just like this time. Could this be related to the issue?
2 hours ago
While I’m not certain if this memory anomaly is the root cause of the issue I'm reporting, I would appreciate it if you could take action to address it as soon as possible.