2 months ago
Everything had been working fine until recently, but after a recent deployment of my Next.js app to production (Southeast Asia🇸🇬), I started experiencing a massive drop in performance and extremely slow network communication.
Even though there were very few code changes in this deployment, the build process took significantly longer than usual. Since I haven't made any major changes on my end, I suspect there might be an issue on Railway's side.
Right now, the performance and network connections remain highly unstable and terribly slow, and this issue has persisted continuously since the deployment.
Could you please look into this to see if there are any ongoing issues or anomalies with the service?
Details of the situation:
・I deployed the app around 3:00 AM today (JST), and the performance has been consistently poor ever since.
・The build took noticeably longer than usual, which already seemed abnormal.
・As I have stated multiple times, I have not made any changes to the code or versions.
・There is a clear, drastic drop in overall performance compared to before the deployment.
・Delays are occurring even on simple static pages, and there is massive latency on the API as well.
・Furthermore, regarding the API (App Router), since it only handles very lightweight processes, the response time used to be consistently under 1 second. However, it now sometimes takes more than 10 seconds.
・The API includes database operations, but I have confirmed there are zero delays on the DB side.
Based on all this, it is highly obvious that there is an issue on Railway's end. Please investigate this properly.
As shown in the 1-week log in the attached image.png, there is a sudden drop in memory right after the deployment. Actually, there are two sudden drops in memory in the image (the most recent one being the one I am referring to), and both of these occurred immediately after a deployment. Notably, the older one (about a week ago) was also accompanied by a drastic performance drop immediately after that deployment, just like this time. Could this be related to the issue?
37 Replies
2 months ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 2 months ago
2 months ago
Hey, this was almost certainly on Railway's side there was an incident on their Edge Network causing elevated latency, with partial outages in US West (California) and US East (Virginia):
https://status.railway.com/incident/A97BS3FC
Since it's been resolved for a while now, trigger a fresh redeploy and see if things go back to normal. If it's still slow after that, it's likely unrelated to the incident worth checking your recent changes or opening a ticket with Railway support.
2 months ago
No, It is the South Asia region.
2 months ago
It is unrelated to that outage.
2 months ago
Ah try this:
Go to your service -> Deployments -> find the last deployment that was fast -> Redeploy it. If:
- Fast again -> the problem is in your new build, not Railway
- Still slow -> platform/infra side, open a ticket
2 months ago
Yes, it remains just as slow. As I mentioned earlier, I made zero changes to my code. The problem is clearly with your service.
2 months ago
FWIW I tested your app from Europe: TTFB is ~240ms after connect, which is healthy for a route through Europe to a Southeast Asia origin. So the origin itself responds fine on this route the slowness is either specific to your network path, or to dynamic/API routes. Can you tell where exactly you see it: page loads in the browser, API responses, or server-side calls to a DB/external service?
2 months ago
When it's working properly, the speed is normal and fast. What can you possibly tell from just a single test? It's taking an unbelievable amount of time.
Attachments
2 months ago
It seems it's sometimes taking over 2 seconds to load just a simple static page. This is absolutely ridiculous.
Attachments
2 months ago
Details of the situation:
・I deployed the app around 3:00 AM today (JST), and the performance has been consistently poor ever since.
・The build took noticeably longer than usual, which already seemed abnormal.
・As I have stated multiple times, I have not made any changes to the code or versions.
・There is a clear, drastic drop in overall performance compared to before the deployment.
・Delays are occurring even on simple static pages, and there is massive latency on the API as well.
・Furthermore, regarding the API (App Router), since it only handles very lightweight processes, the response time used to be consistently under 1 second. However, it now sometimes takes more than 10 seconds.
・The API includes database operations, but I have confirmed there are zero delays on the DB side.
Based on all this, it is highly obvious that there is an issue on Railway's end. Please investigate this properly.
Attachments
2 months ago
I can't pinpoint exactly where the bottleneck is, but it seems like the delay is happening at the routing level.
2 months ago
I am seeing similar delays in the past hour-ish, from US
2 months ago
As shown in the 1-week log in the attached image.png, there is a sudden drop in memory right after the deployment. Actually, there are two sudden drops in memory in the image (the most recent one being the one I am referring to), and both of these occurred immediately after a deployment. Notably, the older one (about a week ago) was also accompanied by a drastic performance drop immediately after that deployment, just like this time. Could this be related to the issue?
Attachments
sheitabrk
FWIW I tested your app from Europe: TTFB is ~240ms after connect, which is healthy for a route through Europe to a Southeast Asia origin. So the origin itself responds fine on this route the slowness is either specific to your network path, or to dynamic/API routes. Can you tell where exactly you see it: page loads in the browser, API responses, or server-side calls to a DB/external service?
2 months ago
As shown in the 1-week log in the attached image.png, there is a sudden drop in memory right after the deployment. Actually, there are two sudden drops in memory in the image (the most recent one being the one I am referring to), and both of these occurred immediately after a deployment. Notably, the older one (about a week ago) was also accompanied by a drastic performance drop immediately after that deployment, just like this time. Could this be related to the issue?
Attachments
2 months ago
As shown in the 1-week log in the attached image.png, there is a sudden drop in memory right after the deployment. Actually, there are two sudden drops in memory in the image (the most recent one being the one I am referring to), and both of these occurred immediately after a deployment. Notably, the older one (about a week ago) was also accompanied by a drastic performance drop immediately after that deployment, just like this time. Could this be related to the issue?
2 months ago
While I’m not certain if this memory anomaly is the root cause of the issue I'm reporting, I would appreciate it if you could take action to address it as soon as possible.
sheitabrk
Hey, this was almost certainly on Railway's side there was an incident on their Edge Network causing elevated latency, with partial outages in US West (California) and US East (Virginia): https://status.railway.com/incident/A97BS3FC Since it's been resolved for a while now, trigger a fresh redeploy and see if things go back to normal. If it's still slow after that, it's likely unrelated to the incident worth checking your recent changes or opening a ticket with Railway support.
2 months ago
May I ask if there are any updates on this?
2 months ago
May I ask if there are any updates on this?
2 months ago
May I ask if there are any updates on this?
constsn
https://www.furima-alert.app
2 months ago
I just tried to visit this page and it is loaded normally
2 months ago
Could you please investigate and explain the root cause of the behavior shown in the image?
Since the application loads normally and quickly depending on the timing, a response stating that "it loaded fine when accessed" is unfortunately insufficient to resolve this issue.
As the attached log clearly indicates, there is an obvious response latency and a significant memory anomaly. I would highly appreciate it if you could review the data carefully and provide a proper, detailed response regarding the actual cause.
・「As shown in the 1-week log in the attached image.png, there is a sudden drop in memory right after the deployment. Actually, there are two sudden drops in memory in the image (the most recent one being the one I am referring to), and both of these occurred immediately after a deployment. Notably, the older one (about a week ago) was also accompanied by a drastic performance drop immediately after that deployment, just like this time. Could this be related to the issue?」
・「Details of the situation:
・I deployed the app around 3:00 AM today (JST), and the performance has been consistently poor ever since.
・The build took noticeably longer than usual, which already seemed abnormal.
・As I have stated multiple times, I have not made any changes to the code or versions.
・There is a clear, drastic drop in overall performance compared to before the deployment.
・Delays are occurring even on simple static pages, and there is massive latency on the API as well.
・Furthermore, regarding the API (App Router), since it only handles very lightweight processes, the response time used to be consistently under 1 second. However, it now sometimes takes more than 10 seconds.」
Attachments
mayori
I just tried to visit this page and it is loaded normally
2 months ago
This is out of the question🫨
2 months ago
Here are the facts regarding the current situation:
・Abnormal memory behavior after deployment: There is a distinct anomaly, yet no specific error logs are being generated.
・Severe latency across all endpoints after deployment: The delay is clearly happening on the Railway server side. I suspect process crashes or restarts may be occurring, though I cannot confirm if this is directly tied to the memory issue mentioned above. Again, no error logs are shown for this either.
・No changes to production configurations, application code, or versions: I have not modified anything for a long time. This exact issue has been occurring immediately after every deployment over the past month, but I am reaching out now because it has become exceptionally severe this time.
Given these facts, are you seriously closing this case simply because "it successfully loaded once"? Isn't it your responsibility to resolve the root cause?
Nothing has been identified yet. Since the issue is almost certainly on your server side (perhaps within the runtime environment?), it is impossible for me to pinpoint the cause from my end. Please investigate this properly.
Attachments
2 months ago
It is true that quite some time has passed since the deployment, and the latency has started to decrease. However, does this mean it is now a new specification of Railway that severe delays will occur for several days after every deployment?
I still want to identify the root cause and find a fundamental solution. The entire behavior remains completely a mystery to me—especially how the performance changes and fluctuates like this as time passes.
Surely you understand that "it is now accessible with less latency" does not mean the root issue has been resolved. I would like to keep this dialogue constructive and work together to find a real solution.
Attachments
mayori
I just tried to visit this page and it is loaded normally
2 months ago
Whether the application is accessible normally at this exact moment is not the main point here. (Of course, it is great that my users can use it without issues right now, but as a developer, I must think about the long-term stability.)
Have you actually reviewed the image logs I uploaded and the issues I described earlier? Given all the evidence I have provided, I cannot understand how you can still maintain that position.
constsn
Whether the application is accessible normally at this exact moment is not the main point here. (Of course, it is great that my users can use it without issues right now, but as a developer, I must think about the long-term stability.) Have you actually reviewed the image logs I uploaded and the issues I described earlier? Given all the evidence I have provided, I cannot understand how you can still maintain that position.
2 months ago
I do not believe this person works at railway
benanthony961
I do not believe this person works at railway
2 months ago
No, I'm just a user just like you guys
2 months ago
Dear Railway Team,
Could I please get an update on this? Are there any findings or responses from your side regarding the issues I raised? I hope this ticket is not being left unaddressed.
Thank you, and I look forward to your reply.
2 months ago
Dear Railway Team,
Could I please get an update on this? Are there any findings or responses from your side regarding the issues I raised? I hope this ticket is not being left unaddressed.
Thank you, and I look forward to your reply.
2 months ago
Does this mean I shouldn't deploy my application anymore??? Please, take some form of action on your end, or at the very least, provide a proper response.
Attachments
2 months ago
Dear Railway Team,
Could I please get an update on this? Are there any findings or responses from your side regarding the issues I raised? I hope this ticket is not being left unaddressed.
Thank you, and I look forward to your reply.
2 months ago
Just on your hunch on the memory, I think that is pretty normal for a containerised application. The running container will naturally hold memory until the garbage collector (at least in the case of .NET) decides to release it. When you trigger a deployment, the container is rebooted and therefore the memory will be dropped in place of the new container.
Do you know whether you have had more traffic than usual to the site?
2 months ago
As I have already stated multiple times, there are absolutely no changes on our end—neither in our code nor in traffic volume. Checking for a spike in requests is the very first thing anyone would do, and that is obviously not the case here.
Dear Railway Team,
Could I please get an update on this? Are there any findings or responses from your side regarding the issues I raised? I hope this ticket is not being left unaddressed.
Thank you, and I look forward to your reply.
2 months ago
It was a simple ask, you had stated there were no code changes but not no increase of traffic. No wonder there is a lack of communication on this thread with how you’re talking to people who are trying to help. Most people that have tried to help you from the community (not Railway employees btw) have been received with snotty responses and witty comments.
2 months ago
Dear Railway Team,
Could I please get an update on this? Are there any findings or responses from your side regarding the issues I raised? I hope this ticket is not being left unaddressed.
Thank you, and I look forward to your reply.
hazzo01
It was a simple ask, you had stated there were no code changes but not no increase of traffic. No wonder there is a lack of communication on this thread with how you’re talking to people who are trying to help. Most people that have tried to help you from the community (not Railway employees btw) have been received with snotty responses and witty comments.
2 months ago
Sorry, but I only want a response from official Railway staff.


