Same here - nothing is getting deployed at the moment

Hi Nick please standby we are investigating, incident has been called

Thanks david, adding some context where I have it in case it helps debugging

We came back online ~30 mins ago. Now we're back offline as of ~4 mins ago

Our apps and services are still down as well, tried migrating to US region, no luck.

Pls help, I can't connect to postgres db any more

Still down for us too.

Do you have backup, I'm thinking of migrate database to other provider

Don't be too hasty – this should be resolved soon (given how long it took last time) though I'm not aware of your requirements. At a certain point that'd have to be an option but for us we won't as yet.

Starting to see our services up now…

Thanks partbot, trying to redeploy but no luck as yet. I'll also check in when we're up

Update: Partial recovery, 50% of capacity restored. Actively working on the rest. Thanks for your patience, on-call team working as swiftly as possible to restore service.

thanks david

Thanks David and team

Still down, i'm trying regularly to re-deploy to no avail

Update: The core issue has been identified and a resolution is in progress to restore service. The on-call team is working to roll it out.

4 out of my 5 services redeployed properly. One more still haven't recovered. Might take a while more for the fix to be rolled out

Thanks for the feedback, acknowledged. That's on me personally for not communicating more. We've had the full on-call team on this (with several additional engineers joining) for as many hours as service has been down.

Full service restoration in sight.

Fix implemented. Resolved.

We've published a full incident retro here: https://blog.railway.app/p/2024-05-04-incident-report