a month ago
Environment:
Region: us-west2
Service: MySQL
Proxy endpoint: [redacted].proxy.rlwy.net:[port]
---
Issue Summary:
Had an error connecting to the database service, thru proxy endpoint.
Production website is down.
All database service dependent features are failing.
---
Issue 1: MySQL Database Public Proxy Errors
Tried connecting to its proxy endpoint, using HeidiSQL and Workbench. Both returned a connection error.
---
Issue 2: Website down
Tried accessing the website to see the changes I've made, but it returned 500 Server Error. When the logs are checked, the error consists of a connection problem with the database.
---
Issue 3: Re-deployment Stuck
Tried to restart the database service, but nothing happened.
So I re-deployed the containers, and it has been 5 minutes, but the deployment hasn't finished. It's stuck.
---
Observations:
Tried connecting to the database service using SSH, but it keeps on saying "Error in server messages: Session error", which is not the case with the Website service, as I got inside the interactive shell of it.
On the Website service metrics, request error rate is 100% as of this writing.
---
Impact:
Website is down. All database-dependent features are failing.
5 Replies
a month ago
I'm experiencing the exact same issue on MySQL in us-west2:
- Database deployment online but unreachable
- Proxy endpoint connection fails
- SSH session error
- Restart / redeploy stuck
This seems to be affecting multiple users and may be an infrastructure issue.
phatnguyen1235
I'm experiencing the exact same issue on MySQL in us-west2:- Database deployment online but unreachable- Proxy endpoint connection fails- SSH session error- Restart / redeploy stuckThis seems to be affecting multiple users and may be an infrastructure issue.
a month ago
I saw this on my notification, a few mins after this question is posted.
I guess this applies to you as well, sir.
a month ago
Thanks for sharing this.
I didn’t get the notification yet, but I’m seeing the same MySQL issue in us-west2
(unreachable DB, proxy/SSH session error, stuck redeploy).
Will wait and monitor.
jose_comparelocalpros
I saw this on my notification, a few mins after this question is posted.I guess this applies to you as well, sir.
a month ago
Quick update: I can connect to the DB using TablePlus now
Railway UI still seems laggy, but the DB itself is reachable.
I’ve backed everything up and moved to another provider just to be safe.
Appreciate you sharing the heads-up!
a month ago
Hello everyone,
Sorry for the late reply, the host that you folks were on was affected by a physical hardware issue that required us to failover the workloads to a new machine. In the meantime, your workloads were briefly inaccessible for this period of time. We are going to be reviewing our notification system to... no longer underplay the impact that happened. I apologize for the delayed reply on my side to get back to you all.
Status changed to Awaiting User Response Railway • about 1 month ago
a month ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • about 1 month ago