(can't hit station support pages either, content never loads)

my entire project and all deployments look like being wiped out lol

We are lookin into it!

Just a note that this is not affecting deployed services.

It affect new deployments I believe

Can you guys log in normally? I'm having issues. This message Failed to fetch was displayed in the form after writing the code I received to log-in.

current deployments are affected

I mean, new pushs for deployed services

"my database is worth more then the pentagon"

yeah because of my workers haha

got my site back up on another server, phew.. luckily it was de-coupled with backend on another cloud provider.. clients websites are still down though.. 🙁

which are eating memory like crazy

misunderstood you

now i'm wondering how much memory are you consuming?

my bad didnt explain it right

didnt check once to be honest but what i check is the bill every month haha

nah I'm Europe based, but one click and i could be up yeah

Once again, why was there no email, notification, anything when this happened? I found out from Twitter an hour after my site has been down - I have incident emails turned on in my account.

alright, can I ask what you do?

collecting crazy amount of data

My bet is that the incident also affected the email sending workers, https://status.railway.com/ is also acting funky because of the incident.

On my side, no twitter, no email, 1 user asking in the general chat and 10 tickets opened ahah

Yikes

heartbreaking

would i be loosing my databases ?

No, user data is not affected

im not having them in metal servers. is that okey? hope you have a backup atleast to restore.

same plan

No updates from GCP?

Day is starting in Singapore, customer complaints keep coming

deleting the memes/imgs let's keep this thread on topic

How do you get blocked from GCP with millions of users on it?

apologies

unbelievable down for hours

Unpaid bills??

No, GCP wrongfully auto-banned Railway's account causing the outage as parts of Railway's infrastructure (particularly their network layer) relies on GCP

only thing I can think of

this should not last for hours🤔

There must be more to the story. kinda crazy it happens on the day of Google I/O. Maybe they are cleaning house

agree, it's like GCP threats railway like a small random account

Unfortunately all we know at the moment, hopefully more is uncovered over time

yeah while 3 million customers are down...

@Fragly you guys see any progress?

Last update was 1hour ago

They just made an update:

Identified

We are continuing to work on restoring access to our workloads on Google Cloud. In parallel, our infrastructure team is evaluating alternative paths to bring affected services back online. We do not have an ETA at this time and will continue posting updates as we have them.

I haven't received any updates since my last update that isn't on the incidents page unfortunately

Should we be worried about our data? I wouldnt expect restoring access shouldnt take more then 4 hours...

Don't worry, data wasn't impacted

any official person here? No eta? Any eta from gcp?

The only impact is GCP hosted components of Railway's infrastructure (like their network layer), data is hosted on Railway's datacenters and or tigris (for buckets)

do you work for Railway?

no ETA, and if by official person you mean Railway employees, Brody is around

I don't, I just help around

@Fragly You don't work for Railway? You helped me a handful of times, I assumed you did.

yea I don't, glad to hear I've been helpful to you 🙂

Is website down?

Can;t log in

https://x.com/Railway/status/2056911420734791875?s=20

yes, there is an ongoing incident, you can track in #🚨｜incidents

still not recover yet?...

We're back!

Some users are reporting partial recovery

oh wow I am back!

mine is back

but don't consider it recovered until recovery is announced

-# saying this out of caution, I've seen users in the past assume recovery and report recovery to their users only for the recovery having been temporary

I am operational! Thank you, Railway. I appreciate your efforts and hope you continue to improve and add redundancy as you grow.

sounds like we had some bad actors possibly reason why GCP banned railway.....

I'm able to log back in, but still missing custom domains and live availability for services. A step in the right direction though

Some,of my services restored

Everything is back up! thanks team Railway

It looks like a Point In Time recovery may have happened, because a ton of my services are showing deployments from 3 weeks ago

The tail end of some services and infrastructure are still coming back online.

😱 what about my database😱

I can access Railway dashboard on my end now!

Most of my databases are showing regular timed backups, but this is my concern as well. Hopefully we aren't missing a BUNCH of data from today

anyone else getting this

and this

On my side, the dashboard is displaying service online, but the service is not reatchable ...

I was having that issue as well, but services are slowly starting to come up for me and very thankful that they seem to have mostly current data.

Also this message in the setting panel:

same

depoyment also not operational yet

Redeploys and some networking still broken for me

Some of Railway's services still need to recovery

Celebrated too early, i've just noticed that not every app is up. Getting "Application failed to respond" errors

Some apps are back, most are still down.

THey are making progress

My app still not reachable but now I can access the dashboard, can see logs, metrics of services etc..

Railway's working on restoring new deployments for enterprise, most metal services should be working now and GCP services are still being brought back online

During those 4 hours of downtime, I lost about ~$1800... I'm a bit pissed off....

That hurts...

is it back up for you?

nope, API is still down

someone else ?

At least you have something ahah

same

all logs are gone in my services

my apps are up now, can access from internet

deploymentn still down

The incident is still ongoing, don't expect full restoration yet

it's alive

Lucky you are!

Where is it hosted (region) ?

US East (Virginia, USA)

Ok thank you!

Any idea on how the recovery is being scheduled?

I have some minor apps back up but some larger ones still down.

Up on my side!

it seems to me that the smaller traffic apps are the first to recover

Still experiencing issues with internal connections to DBs

Makes sense, incident is still ongoing

Both for mysql and postgres

Incident still isn't fully resolved, please be patient

This must be the most stressful day of the year at Railway HQ. Sending good vibes to the team.

Why is Jake tweeting that customers should see full recovery?

Because ultimately they will.

I understand it's hard, but will there be compensation? For example, +1 day to Hobby or Pro Plan

Systems are recovered but backlog is getting hammered with builds, enterprise are prioritised and yours will eventually go through after that.

Well yeah.. obviously eventually. Why tweet something like that implying everything is restored in the middle of the incident?

Internal connections to DB via proxy still down on many services. Any ETA?

No ETA, unfortunately

What is the remaining friction point?

It's most likely part of the incident, the incident is still ongoing

no ETA unfortunately

to be clear, Railway is dependant on GCP's recovery speed for this incident, that's why there's no ETA

man there have been about 5 hours with the my services down

claude pls fix this

asap

Still no updates other than the team is working on it

A lot of services has been restored, there are a remaining few still offline though

main.go:54: [err] Connecting to MongoDB…

That's a mess... on my side API services are up, email service is down!

I am currently losing tone of users registrations because of this outage and thousand of dollars of revenu... I am already lose ~$2000 approx during this 5h outage...

DBs still can't be accessed internally

Some services are still offline unfortunately, nothing we can do but wait

What determines offline vs online services in terms of priority recovery?

Services on GCP vs on Metal.

I have services on Metal which recovered and others which did not

Back online after redeploying DBs

DBs may have been on GCP then.

Metal I think but relieved to be back online

Railway posted a mini post mortem here: https://station.railway.com/community/what-we-know-so-far-may-19th-2026-86354cdd

aNY ADMIN?

If I have a bot on Discord that I can deploy to Railway, can I continue?

Si tengo un bot en discord desplegar en railway puedo continuar?

discord bots are fine (as long as they are not user bots)

Or music bots.

Perfect, thanks for the information ^^

I still can't deploy newer stuff