Skip to content
Services down
pepijn
OP

9 months ago

Multiple of our services are failing or crashing.

Solved

340 Replies

aalfath

9 months ago

same here


9 months ago

Team is aware and looking into it


seanwash

9 months ago

I noticed this in my mysql db's logs:

> 2026-02-11T14:55:06.361425Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user . Shutting down mysqld (Version: 9.4.0).


pepijn
OP

9 months ago

Is this mainly related to internal networking, otherwise I will start rerouting via public network?


9 months ago

It seems related to processes being terminated, team is looking into it, but no harm in trying.


9 months ago

Can you share your service URL? The one that appears in the URL bar (the one that starts with railway.com...) when you have the service in focus.


9 months ago

Seeing this too with my Caddy server. Keeps getting shutdown



alexop1000

9 months ago

Same here, services are refusing to connect


blendibrahimi1

9 months ago

I've tried redeployment, restart, different instances, nothing fixes it


diegogalocha

9 months ago

Same here, the same day we launched a campaign :(


kenchoong

9 months ago

true


9 months ago

image.png


build76

9 months ago

Same here. Services crashing when accessing across multiple sites


seanwash

9 months ago

if I restart my mysql db it seems to work for a minute or two, then it's shut down again


charlie

9 months ago

I'm getting massive response times suddenly, nothing seems to connect to my services either


aalfath

9 months ago

same


kenchoong

9 months ago

all my Rabbitmq instance getting SIGTERM, shuting down


aalfath

9 months ago

this is frustrating


leka74

9 months ago

same issue.


blendibrahimi1

9 months ago

Even railway is having problems, like railway status


lawrencedudley

9 months ago

Also having issues☹networking looks bust and we've randomly got containers that have been terminated.


pepijn
OP

9 months ago

Restart did seem to help (for now)


lawrencedudley

9 months ago

Even this thread is erroring


samueljh1

9 months ago

same problem here.


pepijn
OP

9 months ago

And now dead again


ricardomacario

9 months ago

Same Issue, my production MySQL is down. Affecting hundres of customers


sohcah

9 months ago

Same for my services - restart helped briefly and then died again.


ricardomacario

9 months ago

Same here, we restarted got brief recovery then fully down again


janheussner

9 months ago

Same here


9 months ago

#🚨|incidents has been updated, we just have to be patient while the team fixes it


jorgehenrrique

9 months ago

Same!


blendibrahimi1

9 months ago

Any updatesM


blendibrahimi1

9 months ago

?


9 months ago

None yet, and for updates be sure to look into the incident: https://status.railway.com/cmli5y9xt056zsdts5ngslbmp.


janheussner

9 months ago

I have an uptime kuma app with sqlite db for monitoring and also this is going down. so it looks like it is related to volume mounting.


sohcah

9 months ago

Don't believe so - hitting issues with services without volumes here.


blendibrahimi1

9 months ago

Same here


9 months ago

For reference, the issue seems related to processes being killed randomly, not related to volumes.


aalfath

9 months ago

Any ETA for this?


9 months ago

None at the moment, sorry.


erikskrt

9 months ago

we're having this issue too


Anonymous

9 months ago

all our databases are down and our clients cant acces systems


aalfath

9 months ago

Hope this gets solved quickly... My customers are beginning to notice 🙁


blendibrahimi1

9 months ago

I keep having my services go down almost every month


rafaelmuttoni

9 months ago

Same here, customers are already complaining


blendibrahimi1

9 months ago

Each time i stick with railway cuz i like the service but i can't fucking trust it if it keep going down every month


flschweiger

9 months ago

We're experiencing similar issues with our internal networking when connecting to both MongoDB and Postgres. All instances were running without any problems until approximately two hours ago, when we started encountering connection timeouts.


erikskrt

9 months ago

the status page is also incorrect, doesn't discuss the networking outage


oranuare

9 months ago

Same here!

image.png


9 months ago

It's not related to internal networking.


aalfath

9 months ago

the DB is not even reachable using public URL


seanwash

9 months ago

Is your postgres server actually running though? In all of my services it doesn't seem to be networking related, it's that the services are receiving a shutdown command after a few minutes of being up.


kenchoong

9 months ago

now the services is just crashed randomly

for some reason, it become crashed, just crashed

without any logs


juanqui

9 months ago

This is not acceptable. All our services are hard down right now.


9 months ago

The team will post updates on here: https://status.railway.com/cmli5y9xt056zsdts5ngslbmp. Really sorry for that.


9 months ago

It truly is, again really sorry for that


oranuare

9 months ago

Is not just deployments tho, production went down completely


juanqui

9 months ago

Yeah, the status update is deceptive. All our services are hard down, period.


matthew-cameron1

9 months ago

This is insane, my app is completely offline


kenchoong

9 months ago

ya, the production is went down completely.


erikskrt

9 months ago

fair. i would say that builds failing doesn't fully describe the issue. some existing services went down as well

which is why i saw the status page and assumed it didn't impact us


charlie

9 months ago

I think everything's down, not just deployments


mativm02

9 months ago

all our production services are down :/


matthew-cameron1

9 months ago

Yeah this is not just deployments this is a major service outage effecting my customers


aalfath

9 months ago

It is major outage, never been this bad before


kenchoong

9 months ago

even logs is not readable


erikskrt

9 months ago

yep, had this issue too


fabiensabatie

9 months ago

Lucky you this aint our first time ....


gazhay

9 months ago

I think maybe post an incident in the incidents thread


aalfath

9 months ago

can we at least get some updates? 🙁


aalfath

9 months ago

my customers are getting anxious


seanwash

9 months ago

they did


9 months ago

Can agree here, told the team if they can update it


9 months ago

There's an incident called: https://status.railway.com/cmli5y9xt056zsdts5ngslbmp. Updates will be posted there.


gazhay

9 months ago

Literal seconds after I posted.


9 months ago

A new update has been posted, they're still looking into that.


kenchoong

9 months ago

since June last year, after migrate to the METAL stuff,

it start on and off having the outrage

average 1 month per incident


callmefredcom

9 months ago

Same issue here


9 months ago

From the issues I've been observing, it seems related to the new influx of users to the platform. Been awhile since we got a major outage that affected running workloads (not builds).


samueljh1

9 months ago

i just restarted my MySQL db and it appears to be online again.


aalfath

9 months ago

yeah this one is different, it affects any services, even the already-running ones.


aalfath

9 months ago

please fix it 🙁


blendibrahimi1

9 months ago

A bit more than 1 month ago, all my services went offline, node js mongo etc etc


gazhay

9 months ago

100% not just "deployments"


gazhay

9 months ago

Internal routing is down as notionally my app is up but can't route to the DB. Nothing to do with deployments


9 months ago

Team is trying their hardest to fix it and I can agree that it's unacceptable, they'll be posting a post-mortem on this too.


nunommc

9 months ago

I believe the services have been restored. Please 👍🏻 or 👎🏻 this message if it works for you or not


gazhay

9 months ago

Things happen, as long as it gets fixed and there is an understanding on it and how to avoid it again, all is good - at least from my pov


9 months ago

It's not related to internal networking, the processes themselves seem to be killed randomly.


charlie

9 months ago

Appreciate the Railway team responding quickly!


cben99

9 months ago

What a joke. Responding quickly is not enough. It's one outage a month at this point. Not at all usable.


callmefredcom

9 months ago

Bunch of apps fully down because of this outage.


gazhay

9 months ago

Ah! correct, my db was showing online but clicking into it shows deployment is offline - my node service then can't route to it for obvious reasons!


juanqui

9 months ago

Can you please update the status. "We're investigating issues impacting deployments" is very incorrect. This is a complete outage. It's deceptive to write it this way when it's clear to everyone the impact is far wider than deployments.


jorgehenrrique

9 months ago

O problema aparentemente é apenas no Build com Metal, desabilitei o uso e meus sistemas voltaram a funcionar normalmente, apenas ao desabilitar o Use Metal Build.


kenchoong

9 months ago

been thinking is it worth it for migrate to the METAL thing. before this is just fine. lolz


9 months ago

I've passed this down to the team to update it, they're looking into it!


callmefredcom

9 months ago

I hope their postmortem will be hyper detailed


vinibgoulart

9 months ago

same here


callmefredcom

9 months ago

it has become a major outage

Screenshot_2026-02-11_at_15.31.56.png


sohcah

9 months ago

Railway's postmortems are normally pretty good! Certainly more detailed than many other providers cough AWS cough.


cben99

9 months ago

December 16 outage was exactly the same @ThallesComH 'minor outage' - when everything was down. This is systematic dishonesty.


nunommc

9 months ago

if it helps anybody, my servers are on Legacy, not on Metal! And they're no longer dying every 5 mins.

What GPT had spotted in the logs was that 270GB of physical storage were assigned to the service, so I do believe that as all our projects grow, once in a while Railway will be running out of storage


seanwash

9 months ago

Looks like my services may be back 🤞


kenchoong

9 months ago

u still in legacy? nice.. good for u


9 months ago

I remember Railway having an issue with elevated networking latency, but nothing as critical as this, but I can understand your frustation if that affected your workload, sorry for that.


nunommc

9 months ago

I had to move away from Metal because the connection with Db was lagging a lot


aalfath

9 months ago

Agree this is the worst one in the past 1 year


aalfath

9 months ago

Previously, it was just build failures etc


kenchoong

9 months ago

damn, i still remember that,

the connection is suddenly go for 3s after migrate,

super lagging, damn scary


cben99

9 months ago

I have an email from December 16 saying the following, and it definitely wasn't just build failures, perhaps because it was European time but everything was down

"Hello,

We recently experienced a major outage that impacted your Railway workloads. We're very sorry for the disruption this caused.

Our goal at Railway is to provide a best-in-class experience, and this incident fell short of that standard. To help make things right, we've applied a credit based on your average bill from the last three months (20) to your workspace, Projects. You can view your available credits here.

For a detailed breakdown of what happened and the steps we're taking to prevent this in the future, you can read the full post-mortem here."


waltervito

9 months ago

allow to download database backups, we need an anternative to get back to work


kenchoong

9 months ago

u can go back to legacy after the going METAL


blendibrahimi1

9 months ago

The option is there


blendibrahimi1

9 months ago

I'm trying it


9 months ago

For reference, we run our entire company on Railway and we weren't affected by that issue (to the point of our services going down). I apologize if I gave a different impression, it was not my intention to be dishonest.


kenchoong

9 months ago

really, it dont have that. sad

Screenshot_2026-02-11_at_11.38.34_PM.png


aalfath

9 months ago

can this be changed at least? certainly not caused by the application, it's the fault in hosting provider. 🙁

image.png


9 months ago

Every workload on Railway runs on metal, there's no "non-metal" option.


aalfath

9 months ago

this would provide better visibility for the end users


9 months ago

That's a great feedback, will check with the team!


blendibrahimi1

9 months ago

Idt we even received credits even though all our workspaces were down then


kenchoong

9 months ago

u really no mercy on that.. haha


robertartress

9 months ago

when there's a major outage like this, it would be great to at least be notified by railway so I can at least push messaging to my client notifying paying users about our service outage


vinibgoulart

9 months ago

my services are back now, i'm not using metal, but my apps werent connecting to my database, now they are able


aalfath

9 months ago

yes please, a link pointing to the status page would be great. that will at least mitigate the damage for end users a bit, at least they know that the fault is happening on the hosting provider rather than the developers.


9 months ago

You can subscribe to incidents on and, in case you use Discord, you can follow the #🚨|incidents channel.


pp-tom

9 months ago

We've got clients calling and are pretty annoyed that it's the second time in the past months. It's looking like we might have to change provider in good faith.


blendibrahimi1

9 months ago

It took 10 mins for incidents to be updated with the issue


blendibrahimi1

9 months ago

Even subscribing to it wouldn't do much, clients will probably call before that


codingscape-jay

9 months ago

This is a monthly occurrence now. Your customers deserve a clear write up of what happened, and what steps you're taking to fix it.


9 months ago

I can understand your frustration. Our company also runs on Railway, and we're just as upset as you are about this situation. However, we also recognize that outages are inevitable in any service and we love Railway way too much to leave.


blendibrahimi1

9 months ago

We genuinely don't care if some stuff on your timeline takes longer to get added, we just don't want to have a major outage each month


robertartress

9 months ago

yes, I did that once I found out about the incident. but I only found out about the incident 30min after it started because I happened to open my site to test something


kenchoong

9 months ago

i love Railway very much

just dont outrage every month

i need to sleep


pepijn
OP

9 months ago

Most issues for us seems resolved now after restarts


kenchoong

9 months ago

i actually dont care whether u have update or not

just dont outrage


9 months ago

I can guarantee that the team will prioritize stability over features at any time. For reference, the build issues we were having was their top one priority.


waltervito

9 months ago

Do you have a plan for resolving the problem? Will the data be accessible? Answers are needed; several critical systems are currently down. I need to know if the data is secure.


pp-tom

9 months ago

The benefit to Railway as a developer is amazing, to the client, they just don't want their website to go down and we provide client services so our opinions are less important. It makes it difficult to justify to client why to continue using an unstable service just because I personally like it.


alimardanov

9 months ago

One of my service still not working.


charlie

9 months ago

Is this on the Legacy services or Metal?\


pepijn
OP

9 months ago

Metal, I first restarted database services and then any depending service


daxelrod

9 months ago

Our redis service shutdown and therefor got wiped.


kingman1016

9 months ago

a simple restart fixed it


kenchoong

9 months ago

i still cant see any logs


9 months ago

Can agree here, but I don't think we would have gotten our product to the point it is now without Railway. Every time we don't have to worry about infrastructure is time we get to ship. I can guarantee you that Railway is actively working on improving stability.


flschweiger

9 months ago

This worked for me now, too!


jorgehenrrique

9 months ago

RESOLUTION turn this OFF.

DONE. 🫰

image.png


kenchoong

9 months ago

it work for a while (15 mins)

now 502 again

all service down again


pepijn
OP

9 months ago

I dont think the build environment has anything to do with it


mativm02

9 months ago

it seems to be working again, now I have to deal with a lot of angry customers :|


9 months ago

I don't think that's guaranteed, since some folks (including myself) saw improvement turning it on. In general, I'd use the Metal build environment


hiddenmetrix

9 months ago

database is still down


9 months ago

Have you restarted it?


waltervito

9 months ago

2026-02-11T15:52:23.923886Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11

2026-02-11T15:52:24.081102Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11


9 months ago

Pretty sure that disabling metal build isn't doing much here, as your service received a redeploy which seems to be temporarily fixing the issues for some users. Please keep metal build enabled, it's much faster.


waltervito

9 months ago

i restart , but the mysql doesnt start


blendibrahimi1

9 months ago

I don't doubt that the team is working on fixing issues on Railway but the fact is that major outages are occurring every month


charlie

9 months ago

Restarting seems to have fixed it


hiddenmetrix

9 months ago

ty


bjarki1312

9 months ago

Railway can of course fail, like every service. But at least they hear out and actively fix our concerns. AWS would just put all of us on voicemail...


9 months ago

I can assure you they're aware. We have an internal chat with the team and receive updates on any issues they're actively fixing. One of the examples was the GitHub authentication problem where they told what was happening and the steps they're taking to fix it.


lawrencedudley

9 months ago

The main difference is AWS doesn't go down all the time


kenchoong

9 months ago

this is true actually


lawrencedudley

9 months ago

And when it does it's so bad that our clients are more accepting of it not being our fault


jorgehenrrique

9 months ago

That's what I did in my database and applications, and all 6 had Metal enabled for testing. After disabling it, they all started working again right now. I'm telling you what I did and it worked. What you think based on guesswork without testing is just guesswork.

This can be re-enabled again after the problem is solved, and the problem is with Metal; it's on the resolution page!

image.png


9 months ago

Every workload runs on Railway Metal, which is Railway's own server infrastructure. Service builds were previously running on GCP, and now they're experimenting with running these builds on their own servers as well. Whether your server is built on Metal or GCP should not affect the outage.


9 months ago

Unfortunately we've to wait until the incident is fixed.


waltervito

9 months ago

Excuse me, I don't understand. I don't have any Metal services; they're all legacy, and my MySQL database wont starts. Is anyone else having problems with the database?


tuglabs

9 months ago

A one-hour outage is unacceptable...when will the problem be fixed and will a detailed incident report be released?


callmefredcom

9 months ago

still down in London FYI


kingman1016

9 months ago

restart


9 months ago

Have you run railway services on GCP? Metal's much more stable.


9 months ago

"Legacy" services are still running on Metal iirc, it's just the runtime that's legacy


jorgehenrrique

9 months ago

What I did worked, and I shared it. I'm fine, do whatever you want. For me, it's already resolved. I don't care if it's just guesswork.


kenchoong

9 months ago

still down in Singapore, FYI


jorgehenrrique

9 months ago

Yes, but only the applications on metal were affected. I disabled them until they fix it; it's a temporary solution. It doesn't matter what's best right now, what matters is what's working.


9 months ago

Hey, just got confirmation from the team that redeploying the affected services should fix the issue for some users (this explains why @Gandalf service is working, which is not related to Metal Builds). Just doing a redeployment on each service for example your API and database should fix the issue for some.


9 months ago

Not a great solution, I can agree, but the team is still working on a fix for everyone else


gazhay

9 months ago

I did a simple "restart" of the already deployed SQL service and everything came back for me


9 months ago

Yep, restart also works


callmefredcom

9 months ago

Still down for me


i2gor87

9 months ago

I never used Metal for my database and it's down as well 🙁


alimardanov

9 months ago

I did redebploy. two of my three services are fine. But one is still down


matthew-cameron1

9 months ago

I redeployed however I am still running into 502's with my caddy proxy to my services


marceloqr

9 months ago

after restart mysql service it started working again


safecws

9 months ago

All my apps are down, servers and job applications are out. @Railway address promptly.


9 months ago

Hey, can you try redeploying each of your services as mentioned above? That should fix for some users.


i2gor87

9 months ago

This actually helped my database to go online


callmefredcom

9 months ago

I cant redeploy the database, it would erase it... can only restart it I presume.


seanwash

9 months ago

You don't have a volume attached to your db?


i2gor87

9 months ago

I did restart it

image.png


Anonymous

9 months ago

I am afraid to restart my database! what would happen with my data


malias55

9 months ago

same here. databses down


9 months ago

If you've a volume attached to your service, that shouldn't happen. If you don't know what a volume is, it's this small "card" displayed below your service.

image.png


9 months ago

Hey, can you try redeploying each of your services as mentioned above? That should fix for some users.


i2gor87

9 months ago

for me the data wasn't corrupted or removed. I'm still checking the integrity, but based on what I see - data seems to be okay


Anonymous

9 months ago

this? @ThallesComH

image.png


scanfly

9 months ago

Stuck with development postgres server not being able to be connected to via backend. Prod still up.

Database Connection

Attempting to connect to the database...


9 months ago

Yep, you can redeploy it no problem.


9 months ago

Hey, can you try redeploying each of your services as mentioned above? That should fix for some users.


Anonymous

9 months ago

Ok let's go

image.png

ac2b2aa62baf68e5c4533bb514e1aa27.png


teki-agency

9 months ago

My postgresql service can't be reached


pp-tom

9 months ago

Whoever has a database and the data in a single deployment should be able to SSH in to it right? Though no one should be doing that ever.


matthew-cameron1

9 months ago

Redeployed didn't work but seems services are coming back after


9 months ago

Hey, can you try redeploying each of your services as mentioned above? That should fix for some users.


9 months ago

Maybe, but I don't see why you would need to SSH now though.


ryanhaticus

9 months ago

Just had three client websites crash and the automatic restart policies didn't work at all.


pp-tom

9 months ago

Just an idea but people who use Railway are people who are not comfortable with infrastructure or understand it completely. Railway creates this abstraction and makes it visually easy to understand. But this doesn't mean you can have completely no knowledge of infrastructure like Vercel, It's still involved.

https://svelte.dev/tutorial/svelte/welcome-to-svelte

Svelte has this really nice interactive tutorial... Imagine if Railway had this.


blendibrahimi1

9 months ago

Oh they worked but all the restarts failed


9 months ago

For people still having issues after restart/redeploying, please open a #✋|help thread and I'll escalate it to the team.


andrewchilds

9 months ago

Confirming that redeploying my Postgres and app services brought them back online. Thanks @ThallesComH 👍


ryanhaticus

9 months ago


teki-agency

9 months ago

I'm back online


callmefredcom

9 months ago

Can you please confirm we can safely redeploy a Postgres DB without affecting it?


9 months ago

Railway has an SSH option by using the CLI, just type railway ssh.


9 months ago

Can you share a screenshot of your service on the canvas?


pp-tom

9 months ago

Huh?


hasanmuradi

9 months ago

Is it fixed? Because i can't redeploy


callmefredcom

9 months ago

Do not want to share the name of my services publicly


andrewchilds

9 months ago

I just did and it's fine - the service mounts a persistent volume that survives the redeploy.


9 months ago

You can blur it out but shouldn't be a problem.


9 months ago

or temporarily renaming it to another name


callmefredcom

9 months ago

redeploying the web app part

Screenshot_2026-02-11_at_16.21.05.png


9 months ago

You can redeploy it, no problem.


callmefredcom

9 months ago

Sorry the DB is mysql


callmefredcom

9 months ago

sure? No risk?


9 months ago

Yep


bpsystem

9 months ago

Error de aplicación: se produjo una excepción del lado del cliente al cargar station.railway.com (consulte la consola del navegador para obtener más información) .


hasanmuradi

9 months ago

Help, i can't redeploy

image0.jpg


9 months ago

Hey, can you try switching to using railpack instead of nixpacks? You can change by that going to your service settings


callmefredcom

9 months ago

Taking ages for the DB....


hasanmuradi

9 months ago

Still

image0.jpg


callmefredcom

9 months ago

and still very unstabled on another service which was back up


Anonymous

9 months ago

This is really bad, it's effecting all our services that use MYSQL, tried restarting MYSQL, which seems to fix the problem. Any updates? Please focus on reliability, I really love Railway and don't want to migrate to another platform. Client's are starting to notice that things are not working


arthurl31

9 months ago

can i turn on my redis again already? it was really slow/refusing connection.


Anonymous

9 months ago

But it shows that the service is still online

Screenshot_2026-02-11_at_11.33.03_AM.png


vinibgoulart

9 months ago

yes, just restart/redeploy your services


callmefredcom

9 months ago

It has been more than 10' since I hit redeploy. Aborting and trying again. What a mess.


lawrencedudley

9 months ago

I think it's not


callmefredcom

9 months ago

can't redeploy the mysql DB. stuck


travyo

9 months ago

Something like this is unacceptable, I've been offline for 2 hours


callmefredcom

9 months ago

@ThallesComH Still down, cannot redeploy the DB

Screenshot_2026-02-11_at_16.39.22.png


bpsystem

9 months ago

Ya estoy perdiendo clientes, muy enojados. Que va hacer railway al respecto? ahora como vuelvo a convener a los clientes?

I'm already losing customers, and they're very angry. What is Railway going to do about this? How am I going to win them back?


bpsystem

9 months ago

Ya estoy perdiendo clientes, muy enojados. Que va hacer railway al respecto? ahora como vuelvo a convener a los clientes?

I'm already losing customers, and they're very angry. What is Railway going to do about this? How am I going to win them back?


9 months ago

Hey folks! Can you all try triggering redeploys on your services or anything dependent that's running into this? Say you have an API that depends on Redis and Postgres—try triggering a redeploy on Postgres, Redis, and the API service. That should fix it.

To make sure we get back to all of you, can you please create your own station thread? Want to make sure we miss nobdoy and everyone gets in a good place


callmefredcom

9 months ago

It has been like that for ages

Screenshot_2026-02-11_at_16.42.05.png


9 months ago

For reference, Noah will be replying to this thread from now on, I really need to get back to work, sorry if I didn't reply to someone before!


callmefredcom

9 months ago

Cannot redeploy a mysql DB located here

Screenshot_2026-02-11_at_16.43.25.png


callmefredcom

9 months ago

stuck in Deploying mode


callmefredcom

9 months ago

This is honestly driving me crazy


callmefredcom

9 months ago

I've been here for more than 1h30' now


callmefredcom

9 months ago

Screenshot_2026-02-11_at_16.45.48.png


nchilds77

9 months ago

I am still down. My POSTGRESS DB is un reachable in 2 of my 3 environments. A major client of mine is offline..... Restart doesnt work and a redeploy just hangs..


pp-tom

9 months ago

My services are backup but I'm getting, which isn't true, it's back up. Though that's just a UI thing.

image.png


callmefredcom

9 months ago

Is there anyone who can concretely update us on what's going on?


callmefredcom

9 months ago

My major app has been down for 2+ hours now


hasanmuradi

9 months ago

Screenshot_20260211_194845_com_android_chrome_SameTaskWebApkActivity.jpg

Screenshot_20260211_194818_com_android_chrome_SameTaskWebApkActivity.jpg


hasanmuradi

9 months ago

Anyone can help me i loss users 🙃


leka74

9 months ago

redeploying the services fixed it for me now.


exsnake

9 months ago

Disaster! Everything down for about 2hrs


callmefredcom

9 months ago

cant redeploy my mysql db, stuck in DEPLOYING mode


seanwash

9 months ago

I have a couple services stuck in deploying step now as well


leka74

9 months ago

I dont have a mysql db myself, only a redis service. I managed to redeploy it fine.


callmefredcom

9 months ago

Stuck


callmefredcom

9 months ago

What a commercial nightmare


9 months ago

Ya, lots stuck for me too (stuck ones seem to be ones with a pre-deploy script)


callmefredcom

9 months ago

Screenshot_2026-02-11_at_16.56.20.png


callmefredcom

9 months ago

Is there some official rep from Railway in this thread who can update us on the actual issue?


arananet

9 months ago

im having the same problem, both services are online but unable to communicate each other and there's no log nothing only in the db says

image.png


arananet

9 months ago

ahh good call I just tested and works


arananet

9 months ago

fixed by restart


callmefredcom

9 months ago

I cannot redeploy my mysql DB, stuck


callmefredcom

9 months ago

Can some pls help? I've been repeating the same for the last 30'


arananet

9 months ago

image.png


9 months ago

on "Creating Containers"? That normally means a busy queue from people redeploying after the incident, it will eventually deploy.


arananet

9 months ago

a health prob might be needed to trigger the restart


callmefredcom

9 months ago

Hopefully but that is crazy how long this takes


callmefredcom

9 months ago

Screenshot_2026-02-11_at_17.00.35.png



9 months ago

Yep, sorry for the incovenience, this can happen after an incident, people spam builds and the queue gets crowded


callmefredcom

9 months ago

No idea of an estimated time to resolution?


callmefredcom

9 months ago

it has been 12' since I hit Redeploy on the DB


9 months ago

Please use restart, not redeploy.


alimardanov

9 months ago

It has been more thatn 2 hours


alimardanov

9 months ago

My service is down


callmefredcom

9 months ago

I did it first, did not solve the issue


callmefredcom

9 months ago

I am stuck on redeploying on the mysql DB for 15'


alimardanov

9 months ago

Yeah, i a restarting redebploying for like 2 and a half hours. Still not working


hasanmuradi

9 months ago

Anyone can help me i have pro plan

Screenshot_20260211_194845_com_android_chrome_SameTaskWebApkActivity.jpg

Screenshot_20260211_194818_com_android_chrome_SameTaskWebApkActivity.jpg


alimardanov

9 months ago

Does deleting my service and createing a new instance fix this?


travyo

9 months ago

Restart and Redeploy not working for my MySQL. + 2 hour offline!!!!!


callmefredcom

9 months ago

FAILED after 17'

Screenshot_2026-02-11_at_17.07.38.png


9 months ago

Nope no need to delete.

Can you please select the stuck/offline service, abort any running "redeploys" or "builds" and press "command/ctrl + k" and select "Deploy latest commit" or if a docker image, "redeploy source image"


9 months ago

That will deploy a fresh variation.


9 months ago

can you link?


callmefredcom

9 months ago

as in?


azeemh

9 months ago

restarting the affected services worked for me. Thanks for the fix.


9 months ago

If you select that deploy and copy the URL that is in your browsers search bar that'd be what I'm looking for


9 months ago

So sorry y'all hit this. We'll be providing a very detailed post mortem of what happened


callmefredcom

9 months ago

sent privately. Please update me in DM.



raivandeberg

9 months ago

i tried to redeploy and restart, but nothing happened


travyo

9 months ago

Restart and redeploy not working for MYSQL!

3 hours offline.. bye bye railway


hj-david

9 months ago

Frustrating. When is it going to be fixed?


build76

9 months ago

This may be the thing to push people over the edge. This is the second such outage in 12 weeks. Especially frustrating that Railway have recently raised £100 MILLION dollars Series B funding. I can't build a business around these kinds of failures. Appreciate all Railway are doing to mitigate but this is costing folks livelihoods.


safecws

9 months ago

regardless of the ReDeploy apps dont work. This S4cks. Cannot be happenning. I need to launch something extremely important and you guys are down.


romanmoha

9 months ago

Nothing has been working for three hours now, and this situation occurs consistently once a month. What is the problem? I am losing customers.


nchilds77

9 months ago

Restart and redeploy isnt working.. My clients are all DOWN. This isnt resolved.


kokholm

9 months ago

At least one of my services is still down, and cant redeploy (have tried restart multiple times).

It fails whenever it needs to talk to its Postgress

https://railway.com/project/5dfc53a8-dea9-46af-b0a0-536e51368af3/service/eca68ad2-7ea4-4e75-9918-6597e05b4fca/database?groupId=b412cd06-ef5e-487f-82a1-e12ff2af0268&environmentId=2d2473e5-9346-4932-9ab3-7c2c014f8ba6


9 months ago

People that are still having this issue, please open your own separated help thread and provide affected service links.


build76

9 months ago

Anyone using Directus and tried to redeploy their directus service from a docker image with :latest I ran into a failed migration. Rookie mistake! Redis wouldn't start up, directus wouldn't redeploy.

To fix:

  1. Remove redis from directus variables
  2. Change directus service cache variables from 'redis' to 'memory'
  3. This will remove redis from directus so it doesn't need it to redeploy.
  4. Optionally fix the botched migration. Solution to that below.
  5. When you confirm directus service is up, redeploy redis
  6. When you confirm redis is back up, add back the directus service cache variables. Add back the redis variable too.
  7. Redeploy directus.

Optionally fix any botched migrtations:

  1. You'll need to manually mark the migration as complete in Directus.
  2. Run in psql:
INSERT INTO directus_migrations (version, name, timestamp) VALUES ('20251014A', 'Add Project Owner', NOW());

I did this using railway cli: railway login -> railway link -> project/service/postgres -> psql 'railway postres url'


9 months ago

Hey, can you open your own help thread? I will help you there, I maintain a Directus template.


9 months ago

Or are you just sharing a how-to


build76

9 months ago

Sharing how-to. Sorry if that wasn't clear! And used your template before... great work!


9 months ago

Thanks!


aalfath

9 months ago

is there a postmortem published already?


9 months ago

Not yet.


callmefredcom

9 months ago

Just want to say that despite the understandable anxiety we have all experienced today, I still highly value Railway team members for their availability, seriousness and kindness. In a world with so many nasty figures dominating the news, it is refreshing to deal with people who truly care. Thank you.



9 months ago

@Celengan Babi ^


9 months ago

In my opinion, Railway takes incidents very seriously. From what I’ve seen, this is the first incident in over a year that has affected running deployments. Most other incidents tend to involve slow builds, upstream provider outages, or related issues that don’t impact already running services.

It also seems like the number of instances affected in this incident was relatively small, and the team responded quickly to users.


build76

9 months ago

Think my take from this is that first railway staff never seen to rise once to anyones frustrations, including my own! That has to be acknowledged. Secondly as administrators (new and seasoned) we should be building products that account for infra failures gracefully (at least as far as is possible, if it's down, it's just down). I've made some pretty big changes to a few codebases belonging to high stake clients in the last few hours. Finally I was among the many who paniced (likely because of clients shouting down the phone!) but once I relaxed and disected the problem and thought about mitigation properly afterwards, things settled down. My knee-jerk reaction was to redeploy, redeploy, redeploy. Big mistake. Tech is always always going to fail... at some point. No matter the provider. Question is do you prefer someone who takes shit seriously and apologises and changes policy on the spot to stop it happening or someone who will just tell you to put up and shut up. I'm staying.


build76

9 months ago

Yeh I have a back up version of one project on North Flank. Much more expensive and the DX is pretty complex but it's there if needed. I also seperate my frontend with vercel and netlify to keep concerns separate for public facing websites and some apps. All backend stuff though you can't beat railway for DX, simplicity and price. AWS you'd need a million or two to run the stuff I run! Oooffft!

I'm sure as Railway use their series b funding to expand (what it's there for) we'll see even greater things and more stability.


justus-otundo

9 months ago

Hello Railway Team,

I am writing to request immediate assistance with a critical production outage for my project "Javi Ride." My database service has shut down, and I currently have no live environment serving my customers.

The Issue:

  1. Database Shutdown: My production database crashed/shutdown unexpectedly.
  2. Restore Failure: I have attempted to restore from yesterday’s backup snapshot multiple times. Although the restore process completes, the resulting database is empty and contains no tables.
  3. Data Loss Risk: Because the database is down, I am unable to access today's transactions. I urgently need to know if there is a way to recover the data from today (pre-crash) or if you can investigate why the yesterday's backup is appearing as an empty volume.

justus-otundo

Hello Railway Team, I am writing to request immediate assistance with a critical production outage for my project "Javi Ride." My database service has shut down, and I currently have no live environment serving my customers. **The Issue:** 1. **Database Shutdown:** My production database crashed/shutdown unexpectedly. 2. **Restore Failure:** I have attempted to restore from yesterday’s backup snapshot multiple times. Although the restore process completes, the resulting database is empty and contains **no tables**. 3. **Data Loss Risk:** Because the database is down, I am unable to access today's transactions. I urgently need to know if there is a way to recover the data from today (pre-crash) or if you can investigate why the yesterday's backup is appearing as an empty volume.

9 months ago

Please open your own thread!


medim

Please open your own thread!

justus-otundo

9 months ago

Done


juanqui

9 months ago

Thanks for sharing this. There are a few red flags here that are not addressed and honestly concerns me deeply:

  1. "It was dry run in production and showed correct and accurate abuse identification. Only when turned on, via staged rollout in production, did false positives end up being observed." - This is basically like writing "Dunno, it worked in dev". Perhaps this is coming later, but I am very curious to hear why this was not caught in dry-run.

  2. There's no mention here of a rollback strategy. Was one devised and tested before a changed was rolled out? Was it followed?

  3. [Biggest Concern] The incident report says "<3% of our fleet was impacted during this staged rollout." and also says "then initiated a staged, fleet-wide rollout." and "After the rollout was complete". While I understand that battling fraud is hard, and speed is of the essence, I am concerned with how quickly and broadly this was rolled out. It appears to have been rolled out with no isolation between regions, or any other dimension (all regions were impacted). No bake periods in the rollout? No automated blockers for the rollout driven by metrics/canaries/alarms?

  4. I see the incident report says "Staged rollout, by tier," as an improvement, but honestly, this scares me. This is operational excellence 101. Given Railway's scale, I just assumed OE practices like this were SOP. Reading this makes me question what other OE practices are lacking.

I know you all are very busy, and I don't expect answers here. I do appreciate the work you all do. It really has been a ton of fun working with Railway in my spare time.

Also, thank you for being transparent with your incident report culture. This does help earn trust.

I hope you accept this as constructive feedback.


9 months ago

I just fixed my application by redeploying my backend and restarting all services including the db. Agreed with the above that transparency is appreciated.


justus-otundo

9 months ago

Hevent be helped, and business is running out


justus-otundo

Hevent be helped, and business is running out

9 months ago

Can you share your thread link here?


medim

Can you share your thread link here?

justus-otundo

9 months ago


erikskrt

9 months ago

I also don't understand why this part took so long

CleanShot_2026-02-11_at_14.30.362x.png


erikskrt

9 months ago

think that should be addressed


erikskrt

9 months ago

over an hour to revert


aalfath

9 months ago

Thank you for the update and the transparancy. I really hope this does not happen again in the future.

To be fair, this incident was the 1st major outage Railway had affecting running services. I just wish that it could have been prevented by thorough testing when you guys switched from dry run to live run.

On the postmortem page it says:

After the rollout was complete, engineers noticed the enforcement logic was overly broad in its targeting criteria. Rather than isolating only the intended workloads, the system incorrectly matched certain legitimate user processes, including some databases and application services. As a result, the enforcement system sent SIGTERM signals to legitimate user workloads.

Why wasn't it detected during the dry run? The dry run should have produced exact same output as the live run.


pepijn
OP

9 months ago

Also why is my paid account targeted with anti-abuse config?


9 months ago

Abuse actors have paid accounts too.


Status changed to Solved medim • 9 months ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...