8 days ago
Our production website is currently down, and we are unable to identify the exact cause. This is an urgent production issue.
Our developer investigated the issue and suspects that the problem is related to Redis rate limiting / too many requests being sent to Redis, which may be causing the application or Redis service to become unavailable.
Issue:
Production website is currently down/unavailable.
Our application appears to be generating a large number of requests.
Our developer identified a possible Redis rate limiting / "Too Many Requests" issue.
We are currently unable to determine whether the root cause is Redis, application traffic, rate limiting, connection limits, or another Railway infrastructure issue.
Request:
Could a Railway technical support engineer please investigate our production environment and help us identify the root cause?
We would strongly prefer a phone call or live technical discussion if possible, as this is a production outage and we need urgent assistance to restore the website.
Please let us know if a support engineer can contact us by phone, or if there is another way to get immediate technical assistance.
We can provide logs, Redis metrics, deployment information, and any other details required.
Priority: Urgent – Production outage
1 Replies
8 days ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 8 days ago
8 days ago
Try
1.Restart Services for Immediate Relief:
If your application is locked up due to exhausted Redis connections (zombie connections), restarting will drop them and often restore service temporarily. Go to your Railway Dashboard, select your Redis service, and click Restart. Do the same for your main application service.
How to verify: Refresh your website after the deployment completes. If it comes back online, the root cause is almost certainly a connection leak or inadequate connection pooling in your app code.
2.Analyze Redis Metrics in Railway:
Navigate to your Railway Project > select the Redis Service > click the Metrics tab. Look specifically at Memory (RAM) and CPU. If Memory is at 100%, Redis likely crashed due to an OOM (Out of Memory) error.How to verify: Identifying a maxed-out metric confirms whether you need to increase your service resources or fix a memory leak.
3.Check Logs for Specific Error Codes:
Go to your Application service > Logs tab. Search for exact Redis error messages.
- ERR max number of clients reached: Your app is opening connections and not closing them.
- OOM command not allowed: Redis ran out of memory.
- Connection timed out: The Railway internal network is dropping packets, or Redis is completely unresponsive.