Our main production operations have been down for 30 minutes now because of this.

The Redis service keeps randomly going down.

https://tenor.com/biZTp.gif

Our production Redis instance is still down after numerous deployment attempts.

Do you have any logs that you can share?

Is the deployment itself crashing?

No build/deploy logs being shown unfortunately. Sometimes the deployment itself fails, sometimes it passes but randomly goes offline after a couple minutes. At this time there is an "active" deployment (no logs at all) but the instance isn't reachable. Not sure if it's a state or UI bug.

Have you tried renaming back to the original private network address?

Good idea I'll give it a try after this next deployment fails

I'll see if I can repro this issue

Still no luck after changing back to the original address

I'm seeing a simlar issue (related?) - redeployed a service and now it cannot talk to my Redis instances.

FWIW this Redis instance itself was not redeployed.

Trying to SSH into the Redis instance shows:

📦 Your service's container is not running (status: exited)
🔧 Deploy or restart your service, then try again.

@fred do you get that too?

I can't do that because the service is pending deletion right now, but I imagine I would get the same result as you. The container was definitely not running

Redeploy has fixed for me. FWIW, from my service, my-redis was not resolving. my-redis.railway.internal was, however - may play a factor.

Staff, please just delete the service that was linked in this thread. It's been pending deletion for half an hour now and blocks any other changes from going out. The environment is essentially unusable right now.

@Brody @Noah sorry for the ping but something must be done here

Try on railway Station, the team is there more active

Working on a fix for this now

This bug is so very rare I havent seen it in ~4 months

god bless you noah

The Redis2 instance you had pending deletion has now completed cc @fred

Sorry you got nailed by that, not ideal at all

The nature of the bug causing that deletion issue is so very rare its hard to consistently reproduce. However we did allegedly fix it. Have some things coming down the pipe that should nuke it for good

If I'm able to help otherwise please let me know!

that should do the trick thanks for the post mortem

Absolutely!