Umami postgres DB dead after update and redeployment
ruibinch
HOBBYOP

7 months ago

  • Synced my umami fork to the latest, redeploy starts automatically

  • umami migration starts running in DB

  • DB deploy errors out with these logs:

2025-05-18 14:19:17.637 UTC [4575] LOG:  database system was interrupted; last known up at 2025-05-18 13:41:12 UTC

2025-05-18 14:19:17.930 UTC [4575] LOG:  database system was not properly shut down; automatic recovery in progress

2025-05-18 14:19:17.949 UTC [4575] LOG:  redo starts at 0/BF8DD950

2025-05-18 14:19:18.918 UTC [4575] LOG:  redo done at 0/CCFFFF28 system usage: CPU: user: 0.82 s, system: 0.12 s, elapsed: 0.96 s

2025-05-18 14:19:18.936 UTC [4575] FATAL:  could not write to file "pg_wal/xlogtemp.4575": No space left on device

2025-05-18 14:19:18.941 UTC [2] LOG:  startup process (PID 4575) exited with exit code 1

2025-05-18 14:19:18.941 UTC [2] LOG:  terminating any other active server processes

2025-05-18 14:19:18.942 UTC [2] LOG:  shutting down due to startup process failure

2025-05-18 14:19:18.958 UTC [2] LOG:  database system is shut down

Subsequent restarts encounter the same "No space left on device" fatal issue.

Some other information:

  • I have not touched the deployment in 3 months

  • I see that the postgres DB was automatically upgraded to use Railway Metal - it was working fine before I clicked "redeploy" so that's not the issue

Solved

5 Replies

Railway
BOT

7 months ago

Hello!

We've escalated your issue to our engineering team.

We aim to provide an update within 1 business day.

Please reply to this thread if you have any questions!

Status changed to Awaiting User Response Railway • 7 months ago

Railway
BOT

7 months ago

✅ The internal ticket “No space left on device” after volume migration has been marked as completed.

chandrika
EMPLOYEE

7 months ago

Hi there, just increased your volume size and your DB is back up and running!

Status changed to Solved chandrika • 7 months ago

ruibinch
HOBBYOP

7 months ago

Thanks a lot! Just curious - could I have done anything different to prevent this?

Status changed to Awaiting Railway Response Railway • 7 months ago

ruibinch

Thanks a lot! Just curious - could I have done anything different to prevent this?

chandrika
EMPLOYEE

7 months ago

Of course, anytime!

You could give the volume some breathing room. Your volume was still at the limit as when you were on the Trial plan so the size was small. PG building indexes and cache could cause such a spike to fill up within the DB.

Status changed to Awaiting User Response Railway • 7 months ago

ruibinch
HOBBYOP

7 months ago

Right, I've not paid attention to the volume limits config at all tbh... thanks for the prompt help!

Status changed to Awaiting Railway Response Railway • 7 months ago

Status changed to Solved ruibinch • 7 months ago

Loading...