a month ago
Hello Railway Support Team,
I need assistance recovering access to my production MySQL database without losing any existing data.
Project: believable-courage
Environment: production
Affected service: MySQL
Database engine: MySQL 9
Issue:
MySQL is deployed and shows an online status, but database authentication is failing with:
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
The Railway database interface also reports:
Access denied for user 'root'@'127.0.0.1' (using password: YES)
I verified that MYSQLPASSWORD and MYSQL_ROOT_PASSWORD contain the same configured value. However, authentication still fails using the MySQL client from the service console.
The persistent volume is mounted at /var/lib/mysql and contains existing database files. Approximately 226 MB is currently used.
A backup named Pre-Security-Patch Backup (245 MB) is available, but I have not restored it because I want to avoid overwriting newer production data.
Request:
- Please investigate why MySQL authentication is failing.
- Help restore database access without deleting or overwriting existing data.
- Confirm whether the current persistent volume and its data can be preserved.
- Provide a safe recovery procedure before any potentially destructive operation.
This database supports a live tournament management application, so preserving the existing records is essential.
Thank you for your assistance.
4 Replies
a month ago
If the password variables were changed or regenerated at any point, the database files on your volume may still expect the password they had before. Every deployment keeps a snapshot of the variables it ran with: open the MySQL service, click a deployment from before the access errors began, and expand the variables section on its Details tab to see the MYSQL_ROOT_PASSWORD and MYSQLPASSWORD values that deployment used.
If an earlier value is different, try logging in with it. Reading those values changes nothing, and your volume and its data stay as they are. Holding off on restoring the backup until you have regained access is the safe order, since a restore replaces the current data with the backup's contents.
Status changed to Awaiting User Response Railway • about 1 month ago
a month ago
Hello Railway Support Team,
Thank you for your response.
I reviewed several historical MySQL deployments, including the deployment from September 19, 2026, and confirmed that MYSQL_ROOT_PASSWORD has the same value as the current deployment.
MYSQLPASSWORD also references MYSQL_ROOT_PASSWORD.
Unfortunately, authentication still fails with ERROR 1045 (28000), both from the MySQL container console and the Railway database interface.
The MySQL server starts successfully, and the persistent volume still contains the existing database files.
Could you please provide the next safe recovery steps to restore database access without losing or overwriting any production data?
I have not restored the backup or modified the database files.
Thank you for your assistance.
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 1 month ago
a month ago
Try this:
- Remember the current Custom Start Command
- Change the Custom Start Command to
docker-entrypoint.sh mysqld --skip-grant-tables --skip-networkingand redeploy - Go to the console tab of your MySQL service
- In the terminal, run
mysql -u root - Run the following commands:
ALTER USER 'root'@'localhost' IDENTIFIED BY '<PASSWD>';
ALTER USER 'root'@'%' IDENTIFIED BY '<PASSWD>';
FLUSH PRIVILEGES;Where <PASSWD> is replaced with the new password (From the variable MYSQL_ROOT_PASSWORD)
- Revert the custom start command and redeploy MySQL
- Make sure to restart other services that depend on your MySQL database
a month ago
AI-assisted; tested against an isolated synthetic MySQL 9.7.2 database, not your volume. Your exact MySQL 9 minor version and image are still unknown.
There is one correction to the SQL sequence above: with --skip-grant-tables, ALTER USER is initially disabled and returns ERROR 1290. FLUSH PRIVILEGES must come before ALTER USER, in the same recovery session. MySQL documents this order in its recovery instructions.
Preserving your current data comes first. Have Railway support establish a current recoverable backup/snapshot and maintenance procedure before changing the live service; do not restore the older backup over newer records or delete/reset the volume. Do not run a second MySQL server against the same data directory. A password recovery changes grant tables; it is not a read-only operation.
If you and support proceed with the network-disabled recovery mode already discussed, inspect the actual account names first, without retrieving password hashes:
SELECT User, Host, plugin, account_locked
FROM mysql.user
WHERE User = 'root';Then the documented order is:
FLUSH PRIVILEGES;
-- Only if the preceding SELECT showed this exact account:
ALTER USER 'root'@'localhost' IDENTIFIED BY '<your-new-private-password>';Choose and enter the real password privately; do not use or publish the placeholder. Only alter other host entries if they actually exist and are needed. root@localhost and root@% are separate accounts: blindly altering a missing root@% returns ERROR 1396. Do not create wildcard root access just to make that command succeed.
Afterward, remove the temporary recovery flags, restore the original start command and restart normally. Verify a fresh authenticated connection and application access. Merely reaching an online server or a recovery-mode console is not recovery success. Keep the application credentials aligned with the stored password; the initialization environment variable does not overwrite an already initialized database's credentials.
Reproduction and checks: pinned official image, no network or exposed ports, one synthetic row, and one server at a time. The test reproduces both error codes, verifies the corrected order, rejects the old credential, and checks the row and new credential after normal restart. Actual Docker CI.
This corrects the recovery sequence; it does not establish what originally caused your 1045 error or prove your production volume's condition. If you tried the earlier sequence, did it stop at 1290, or does 1045 remain after a normal restart? Please share only the error code and MySQL version, not passwords or private logs.