Hey there! We've found the following might help you get unblocked faster:

• 🧵 Postgres crash: volume mount error failed to exec gzid

• 🧵 Container failed to start issue

• 🧵 Container failed to start - volume mount error

If you find the answer from one of these, please let us know by solving the thread!

Also here

This is extremely urgent, every DB deploy is failing, our entire business utilizes this DB.

Our service has now been down for an hour, do I need to upgrade to an enterprise plan or something in order to get this fixed ASAP

Hey there Ben, not good, jumping on the thread to make sure you are in a good spot.

Gathering context, one second.

thank you

Is there any sort of immediate fix or anything, I can't be waiting a full day to get the service back online

Engaging the platform team here while we look into the issue, will give you realtime updates. In the meantime, I am now going to try to recover access to the DB. It looks like an issue with mounting.

We aren't going to leave you in the lurch for that long.

Ok thank you, that bot message was scary

Yep, the bot message was anything when we attach a Linear ticket to it.

Working through the issue now, going to take a few minutes to debug.

same for me

same here. everything stopped. service offline. couldn't connect to the database. It's been an hour and no signs of recovery

benshaener, your database is online and accessible now.

For anyone else, please provide direct links to your database or service in question.

This is my service link https://railway.com/project/52b44fed-61b5-47d4-90cd-58ad55a0b2cf/service/4f4ec0c6-40aa-4f0c-8a4f-f8a03668bec0/database?environmentId=61adf75b-1eea-4bf1-9e80-5d41c38d628f

Ok, thank you. I am getting only 502 Errors when I hit my API currently, this might be an issue on my end so I'll look into it and let you know if there is any issues with that aswell, thanks

This is mine, I'm having the same issue

https://railway.com/project/12729143-7dd3-4121-9637-b9edc1637a5f/service/9149fae5-7ef5-4d0b-91e5-6cd853eae135/database?environmentId=92d75e8d-5215-46b7-88f4-cc93111d8e0f

We're still working on a fix which should resolve this. Are you able to try redeploying that postgres service to see if it has the same problem?

I had a redeploy before for about 12min. Now I'm trying again, is going on for 4min now

Ok, so the deployment is up now, but I get this when I try to access the database:

Database Connection

We are unable to connect to the database via SSH.

psql: error: connection to server at "localhost" (::1), port 5432 failed: Connection refused Is the server running on that host and accepting TCP/IP connections? connection to server at "localhost" (127.0.0.1), port 5432 failed: Connection refused Is the server running on that host and accepting TCP/IP connections?

this is what I see

My system is back up, but the performance is very significantly worse, like 1 minute wait times worse, its very bad. Don't expect it will stay this way, right? Any sort of estimate of when I can expect it to be back to normal performance?

No estimate yet, working on getting everyone back and then we can address the performance.

I'm getting this in the logs right now

The coalition version mismatch shouldn't be impacting much as its just a warning. You can sort that out whenever you'd like.
As Angelo had mentioned we're currently working towards getting everything back up and performant again!

https://railway.com/project/52b44fed-61b5-47d4-90cd-58ad55a0b2cf/service/4f4ec0c6-40aa-4f0c-8a4f-f8a03668bec0?environmentId=61adf75b-1eea-4bf1-9e80-5d41c38d628f&id=81daa3b3-2046-41bd-89cc-f7e8c5c70e16#deploy

I tried to redeploy but still hanging on creating container

my database now doesnt start https://railway.com/project/e376bd62-88bd-4f57-a11f-81b05faec1b3/service/b7e571f2-529d-476c-bdc1-8417ee72888e?environmentId=e9a18632-cc06-4d0f-9a32-3f38e95ec70c

It has now been over 5 hours since our critical database failure began, and the issue remains unresolved.

We are still completely offline. We have seen no recovery of the service, and the lack of a concrete solution or timeline is alarming.

Every hour of downtime is causing significant damage to our business. We cannot wait any longer.

This situation has escalated beyond a technical issue. Our customer support team is now completely overwhelmed.

Because of this 5+ hour downtime:

Our support channels are inundated with angry customers.

We are facing a massive backlog of complaints that we cannot resolve because the database is unresponsive.

Our brand reputation is suffering irreversible damage every minute this continues.

Completely understand and I am very sorry to hear about this. We're still working towards a fix involving startup times/performance and will let you know as soon as we get info.
This degraded performance issue seems to only be related to our US-East region so if it works for you temporarily switching to US-West should resolve things.

Acknowledged, but the issues I see on the projects that you have attached to the account doesn't show any database or impact related to the database issue unless there is another thread I am missing here.

That said, happy to be the upstream blame for the customers who are reporting that issue.

Separate thread, noting that all the DB reports are back online. Next step for us is to check the performance of the underlying host.

I tried moving to us-west, and now it got stuck. Can I cancel the redeployment safely? It seems to be back again on us-east

Hey Oranuare for the sake of trying to help you out, are you able to make your own thread and link it here? Would love to help debug there.

Yes, I have one called "How long are database backups supposed to take?". We could continue there

Would be ideal if you can create a new one directly for this issue!

Bit of an update for y'all,
Everything has leveled out on our end, and things are back online. We'll keep monitoring this in case something comes up, but for now, everything should be back to normal. Feel free to let us know if you see this again.

✅ The ticket Setup issue with integrations has been marked as completed.

✅ The ticket Setup Issue with Connections has been marked as completed.