Hello,

It's been a series of cascading issues that have hit us over the past few weeks: GitHub API rate limits backing up webhooks, build queues spiking, a false positive in our abuse detection system, and hostile traffic impacting our shared proxy layer.

Any one of those alone is a bad day; stacked together, it was unacceptable. We're sorry.

Since then, we've shipped a lot: staged regional rollouts so infrastructure changes don't hit everything at once, expanded our compute fleet across multiple clouds, rewrote our abuse detection system with proper safeguards, hardened our network with new interconnects and per-container firewalls, and are rolling out a WAF for all customers.

What are you doing for customers dealing with this? I’m constantly getting alerts and having to switch over to backups.

We're focused on the stability improvements mentioned - staged regional rollouts, expanded compute fleet, rewritten abuse detection, hardened networking, and the WAF rollout. The team is all hands on deck making sure these issues don't recur.

Ok but I’m paying for uptime not for you to work on figuring out normal scaling and business issues. Why aren’t credits being granted for issues I have to deal with because your service isn’t stable.

Happy to add credits for the inconvenience we have caused you. I've added credits equal to your last invoice.

Thank you. I just got an alert it's down again!

Can you share which service is down and what you're seeing?

Got things working again after shifting to backup host for an hour or so, may have actually been a load balancing issue from a railway hiccup or ssl refresh.

I'm a huge railway fan, just moved my 3rd app over to it today. I hope this gets sorted out the first month was incredibly smooth and now the daily alerts and today customers reach out for downtime since load balancing to backup didn't go smoothly. It's been a rough week, I'm sure you all are working through it and thank you for the speedy replies. Let's hope tomorrow goes better 🤞🏼

Thank you for the kind words; they are much appreciated.

I'm hosting on railway for prod users and we're seeing multiple of these up/down per day, minutes to hours at a time. It's not reliable enough to be worth paying for. Will be migrating away out of railway, unfortunately.

Hi do you have a product marketing contact that might be interested in a case study or customer story? Things are back to normal mostly this week and I figured a story of an app moving from Replit to railway to get scale and stability could be interesting.

Hi there,

We aren't doing case studies at this time, but we appreciate the offer and are glad things have stabilized for you!

Best,
The Railway Team