f5925531-2de0-4da9-8a6d-15b5ba712ebe

I'm getting this too

P.S i'm in the Pro plan

Same

Also after redeploy, app is suddenly not reachable anymore 😦

not even an error page, just no server response at all

Yeah there's definitely an incidence happening. Hope the railway team sees this soon and starts taking action.

Yep I hope so too ^^

But it's very weird as it only affects my prod and not the testing environment

team has been made aware

If it helps the investigation, my uptime monitor fired at 9am

Ack

Looking into it

good luck!

An incident has been reported -

thank you Brody

Adding to the data, our app is down too, seems like it’s Postgres given that landing page is fine, just main requests from DB are 500ing

the team have not posted an ETA yet

They will be publishing a post mortem after the incident has been resolved, either me or someone else will link it here when it is posted

hey @Brody , would upgrading to pro right now help ? Or will it be the same scenario

Majority of instances should be restored at this point

Experiencing the same issues mentioned above

Some of the projects run fine, others stopped working

Pro users have also been affected

I'm still down

I guess the Database are down.

this incident can affect all services not just databases

you may have to redeploy affected services

I get this error when I do that

I'm getting the screenshot above too, does that mean no deployments?

It means priority deployments first

in the post mortem it would be good to look into why this was first raised on the forum and not via automated monitoring

Yes, Pro users would have builder priority at this time

I'm in Pro plan, tried redeployed but I'm stucked at this:

please note that Jake said a majority, this issue has not been fully resolved yet

It was raised via automated monitoring

But yes, sure

? there was no incident when we riased it here, it started about 9am

Redeploy stuck on Waiting for build to start…, this is a pro plan not hobby

Initially tried restarting the container but didn't seem to do anything

Still unable to deploy services normally…

^

My app instance has been down all this time.

But I was able to access my MySQL instance. Now I can't.

I assume this is part of the restart.

same here. my production site has been and it costs me and customers too 😦

It is what it is.

I owe someone who is not technically knowledgeable an explanation.

hope it gets back up and running again

Limited Access - Disabling for hobby while we restore systems <:HAHAHA:786868887441375243>

I also experience the same issue. Hope things get back to normal very soon

Saying poor in diplomtic

Same here…

I'm back online

My website is still down. But now I get a différent message. It has to do with the database. I can't access the database. I get an error

I get application failed to respond error

On the client

Systems a

Systems partially work again. Live system running, Test system still no database.

@André did you have to redeploy or restart?

@rickitan
For Live to work again I redeployed every 5 minutes to check. Now it works again.
For stage I did nothing yet but the mysql server for laravel crashed. Redeploys / Restarts not woroking for stage yet

Now all systems back to normal

Awesome!!

They are not.

Yes I was asked to restart but deployment still crashes

Same thing here

I think my API are back to normal now

I've done a re-deployment, and back online, thanks, team.

2+ hours is not a good thing though, it's going to hurt.

@King Jahad
Yes but I expect an report to be published hopefully today / tomorrow.
It's one of the most awful thing that can happen. Let's give them some time to investigate and for every operation / downtime costs on client side there is time to discuss after.

The most important thing is to get everything up and running again 😉

<:blobyes:1149662717213417565>

+1

I am not mad, just don't know who to blame for this.

and how to make a client understand

In case anyone is still experiencing issues, make sure you try to restart / redeploy if you're still experiencing issues, hobby or pro

I've tried dozens of times. Still down.

This is the first time it worked for me, shouldn't long now if I had to guess

I couldn't connect to Postgres for the longest time and it finally came online

I had issues with both a hobby and pro service

I can't even access the database from the ui

Try force refresh (shift-command-r)

Same thing.

i'm not able to see the list of deployments, so i can't redeploy what i can't see

I'm back online

I tried to redeploy again and it works now

Thank you very much for your suggestions

i'm seeing the deployments.. so i'm redeploying now

My app still can not redeploy because the connection to Postgres DB still failed 😦

Give it time, same was happening to me.

I believe they are doing a massive restart. So some servers are restarted and come online before others.

for (server in servers) {
await server.restart()
}

Something like that

redeploying my app failed, but it just gives me "no build logs found for deployment"

after several attempts, my server was succesfully redeployed. it wasn't able to connect to an internal Redis

The redeployment was successful, I almost lost my job today. <:HAHAHA:786868887441375243>

I setup demo for my team today and Railway failed just 10mins right before the meeting. Nothing more embarrassed for me than this.

ironically my dev environment is working fine

We're back online too

latest deployment seems to have stopped in the middle of it.. never got to the health check

It was the same for me ^^
At least the "least" important project didn't worked. Suuuureeee 😂

probably time to move to aws 😭

redeploying 3rd time's the charm

i am on beta V2 of the builder, and V2 runtime.. hopefully that is not biting me in the ass

not able to reach the database right now during the deployment

i'm seeing the postgres service, but when i click on data it can't establish a database connection

not seeing my deployments again

same to me, database it's working, but the app it self doesn't work

If I moved to AWS I would probably cause worse downtimes than trusting the railway team lol. It's just not my expertise.

But this 2h long one was definitely a bad one.

the same happened to me on the previous massive outage 😫