a day ago
Hello.
Recently I experiance slow responses on the backend side. Was happening with 1 replica and now with 2 api replicas.
It starts after some new deploy.
No amount of time was fixing it. Only redeploy did.
Initially I though it might be not locked versioning so I locked it in dockefile.
But still it happens. Currently in this deployment one of the replica reponds normally, but other one is about 3x slower.
In extreme cases (deployments), responses were even 10-30 times slower (p99).
Currently I suspect noisy neighours.
I do not know how to confirm this theory.
Obviously redeploying every time I spot slower responses is not a very convenient way and does not always succeds right after, so sometimes it takes 2-3 redeploys to fix the issue.
1 Replies
a day ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 1 day ago
2 hours ago
Yes, this is likely a noisy neighbor issue, and the fact that redeploying (not restarting) fixes it confirms it, redeploy moves you to different host hardware, so the slow replica gets away from whatever's hogging that node.
To confirm: shell into the slow replica and check CPU steal time:
cat /proc/stat | grep '^cpu '
or top/mpstat -P ALL 1 5 and look at the %st column. High steal time (sustained >10%) = confirmed noisy neighbor. Compare against the healthy replica at the same moment, if only the slow one shows steal, that's your proof.
There's no way to pin to a specific host or avoid this proactively on shared compute, take the steal-time evidence to Railway support.