Outage killed my prod db
victorgoic0
PROOP

24 days ago

ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)

Here is the error message I see. Originally, the database connection simply wouldn't establish. I then had the support agent restart the service, which now gives me this error.

Details:

  • mySQL database running in railway
  • node API deployed to railway was working before the outage
  • connections from the node app are all timing out -- connection is impossible
  • certain env vars on the mySQL instance are coming up as blank strings
Solved$20 Bounty

Pinned Solution

0x5b62656e5d
MODERATOR

23 days ago

Try this:

  1. Remember the current Custom Start Command
  2. Change the Custom Start Command to docker-entrypoint.sh mysqld --skip-grant-tables --skip-networking and redeploy
  3. SSH into your MySQL service (Right click the service and Copy SSH Command) (You'll need Railway CLI installed for this to work)
  4. In the terminal, paste the SSH command and run mysql -u root
  5. Run the following commands:
ALTER USER 'root'@'localhost' IDENTIFIED BY '<PASSWD>';
ALTER USER 'root'@'%' IDENTIFIED BY '<PASSWD>';
FLUSH PRIVILEGES;

Where <PASSWD> is replaced with the new password (From the variable MYSQL_ROOT_PASSWORD)

  1. Revert the custom start command and redeploy MySQL
  2. Make sure to restart other services that depend on your MySQL database

12 Replies

Railway
BOT

24 days ago

Your MySQL service is running and ready for connections as of 16:44 UTC. The "access denied (using password: NO)" error confirms your observation that credential env vars are resolving to blank strings, which is a side effect of the ongoing platform disruption. Please check the Variables tab on your MySQL service to verify that MYSQLPASSWORD, MYSQL_ROOT_PASSWORD, and MYSQL_URL are populated. If any are blank, the variable resolution is being affected by the current incident, and repopulating them (or waiting for full platform recovery) and then redeploying both services should restore connectivity.

Status changed to Awaiting User Response Railway • 24 days ago

Railway

Your MySQL service is running and ready for connections as of 16:44 UTC. The "access denied (using password: NO)" error confirms your observation that credential env vars are resolving to blank strings, which is a side effect of the ongoing platform disruption. Please check the Variables tab on your MySQL service to verify that MYSQLPASSWORD, MYSQL_ROOT_PASSWORD, and MYSQL_URL are populated. If any are blank, the variable resolution is being affected by the current incident, and repopulating them (or waiting for full platform recovery) and then redeploying both services should restore connectivity.

victorgoic0
PROOP

24 days ago

They are not populated. And I cannot access the database to see what the values are.

Status changed to Awaiting Railway Response Railway • 24 days ago

Railway
BOT

24 days ago

Your MySQL data is intact, as the volume is mounting correctly and InnoDB initialized successfully on each restart. The blank credential variables are a side effect of the ongoing platform disruption. You can regenerate your MySQL password using the Credentials tab in your MySQL service's Database view, which will set a new password on the database and update the environment variables in sync. After regenerating, redeploy any services that reference those credentials.

Status changed to Awaiting User Response Railway • 24 days ago

Railway

Your MySQL data is intact, as the volume is mounting correctly and InnoDB initialized successfully on each restart. The blank credential variables are a side effect of the ongoing platform disruption. You can regenerate your MySQL password using the [Credentials tab](https://docs.railway.com/databases/database-view#credentials-tab) in your MySQL service's Database view, which will set a new password on the database and update the environment variables in sync. After regenerating, redeploy any services that reference those credentials.

victorgoic0
PROOP

24 days ago

I CANNOT access the credentials tab. It will not connect. The credentials were wiped by the outage. I need a human assistant

Status changed to Awaiting Railway Response Railway • 24 days ago

sam-a
EMPLOYEE

24 days ago

Apologies for this canned message but in an effort to help all our customers get back up and running, we are sending this bulk message. As you may know, we had a major interruption to our services yesterday. We've published a post-mortem if you'd like more information on the incident. It describes what happened and what we are doing to prevent it in the future. We are deeply sorry for the impact that it has had on you.

It is taking some time to bring everything back up, but we are working on it as fast as we can. In general, a redeployment should fix most service issues. Due to the volume of customers redeploying right now, builds and deploys may take longer than normal to process.

You can track recovery status here: https://status.railway.com/incident/KVZ1Z8GY

If you are still having other issues that might be related to the incident you can read more here: https://station.railway.com/community/road-to-recovery-post-gcp-outage-builds-d362e48c

Feel free to respond if your question has not been addressed.

Status changed to Awaiting User Response Railway • 24 days ago

sam-a
EMPLOYEE

24 days ago

Apologies for this canned message but in an effort to help all our customers get back up and running, we are sending this bulk message. As you may know, we had a major interruption to our services yesterday. We've published a post-mortem if you'd like more information on the incident. It describes what happened and what we are doing to prevent it in the future. We are deeply sorry for the impact that it has had on you.

It is taking some time to bring everything back up, but we are working on it as fast as we can. In general, a redeployment should fix most service issues. Due to the volume of customers redeploying right now, builds and deploys may take longer than normal to process.

You can track recovery status here: https://status.railway.com/incident/KVZ1Z8GY

If you are still having other issues that might be related to the incident you can read more here: https://station.railway.com/community/road-to-recovery-post-gcp-outage-builds-d362e48c

Feel free to respond if your question has not been addressed.

victorgoic0
PROOP

24 days ago

Yeah I redeployed the API and the database several times. Still down.

Status changed to Awaiting Railway Response Railway • 24 days ago

mykal
EMPLOYEE

24 days ago

Apologies but this looks like an issue with the application level code; after investigation it looks like the MySQL username and password variables were set to empty strings which is the root cause.

Due to volume, we can only answer platform level issues.

I've made this thread public so that the community might be able to help with you query.

Status changed to Awaiting User Response Railway • 24 days ago

Status changed to Open Railway • 24 days ago

0x5b62656e5d
MODERATOR

23 days ago

Try this:

  1. Remember the current Custom Start Command
  2. Change the Custom Start Command to docker-entrypoint.sh mysqld --skip-grant-tables --skip-networking and redeploy
  3. SSH into your MySQL service (Right click the service and Copy SSH Command) (You'll need Railway CLI installed for this to work)
  4. In the terminal, paste the SSH command and run mysql -u root
  5. Run the following commands:
ALTER USER 'root'@'localhost' IDENTIFIED BY '<PASSWD>';
ALTER USER 'root'@'%' IDENTIFIED BY '<PASSWD>';
FLUSH PRIVILEGES;

Where <PASSWD> is replaced with the new password (From the variable MYSQL_ROOT_PASSWORD)

  1. Revert the custom start command and redeploy MySQL
  2. Make sure to restart other services that depend on your MySQL database

mykal

Apologies but this looks like an issue with the application level code; after investigation it looks like the MySQL username and password variables were set to empty strings which is the root cause. Due to volume, we can only answer platform level issues. I've made this thread public so that the community might be able to help with you query.

victorgoic0
PROOP

23 days ago

Just to be clear, the error I presented is from trying to access the database tab in the railway dashboard. It is not from my application code. The outage has wiped my password and dbname.

0x5b62656e5d

Try this: 1. Remember the current Custom Start Command 2. Change the Custom Start Command to `docker-entrypoint.sh mysqld --skip-grant-tables --skip-networking` and redeploy 3. SSH into your MySQL service (Right click the service and Copy SSH Command) (You'll need Railway CLI installed for this to work) 4. In the terminal, paste the SSH command and run `mysql -u root` 5. Run the following commands: ```sql ALTER USER 'root'@'localhost' IDENTIFIED BY '<PASSWD>'; ALTER USER 'root'@'%' IDENTIFIED BY '<PASSWD>'; FLUSH PRIVILEGES; ``` Where `<PASSWD>` is replaced with the new password (From the variable `MYSQL_ROOT_PASSWORD`) 6. Revert the custom start command and redeploy MySQL 7. Make sure to restart other services that depend on your MySQL database

victorgoic0
PROOP

23 days ago

Will try this right now thank you

victorgoic0
PROOP

23 days ago

I have restored the environment. I highly doubt I will be the only person with this issue, so let me explain to you what I saw from Railway's behavior.

  1. When I tried to copy the MySQL root password and the MySQL password, I noticed they were set to different values.
  2. I copied and saved both, then SSH in.
  3. Considering the commands provided seem to be referencing only one password, I chose to use only one of the values that I had saved.
  4. After successfully altering them, I go back to the dashboard.
  5. The env vars in the dashboard are now NEW values! (not the ones I just copied -- redeploying reset them).

So before fixing the issue, I hardcode the value I had saved to both env var fields, ignoring the secret() call they are set to by default. This is also the point where I noticed that the MYSQLPASSWORD directly references the root password, so the fact that they were different to begin with was already a problem. And again, returning here after deploying, had now set them to two totally new values.

  1. I hardcode both to the value I had saved
  2. I test again.
  3. "No database selected" error in API logs, huh?

It is at this point that I now see that the MYSQLDATABASE environment variable has also been set to a blank string.

  1. I set this to "railway".
  2. I SSH in again to make sure that I'm not making that up.

OK cool, restore old command, redeploy API, and try to test API again. I can now see my tables in the database tab!

  1. Failed, no database selected.

It is at this point that I now realize that my API's database URL is not including the database name. This is despite it being set to railway's MYSQL_URL env var.

  1. I hardcode it to the correct value.
  2. Redeploy and test again.

Finally, our app is working. Anyways so long story short, it looks like secrets management for our services is messed up. We'll just hardcode values from now on. You will probably see this elsewhere.

Thank you for the suggested fix and have a good one!

Status changed to Solved victorgoic0 • 23 days ago

Welcome!

Sign in to your Railway account to join the conversation.

Login
Loading...