2 months ago
All deployments were stopped at exactly 16:58 IST on Jan 25.
I believe this was due to a failed attempt to upgrade to Pro from a Hobby plan but the transaction didn't go through. I then decided against upgrading. I received an email that said the invoice was due (at 16:58 IST). I ignored it because I didn't want to upgrade from the Hobby plan.
Here's the important thing. I was and still am on the Hobby plan.
Despite this, all my deployments were stopped. Why was this the case?
This seems to be a bug in how the billing system works. Since I was on the Hobby plan, none of this should have affected my deployments.
There were some very important websites live. What made it even more confusing was that the Project view shows everything online. Until I went into the project I didn't realize it's offline.
Railway team, need some clarification and resolution here.
6 Replies
2 months ago
Looking at your account, I can confirm you're on the Hobby plan with an active status. Here's what the billing records show happened at exactly 16:58 IST (11:28 UTC) on January 25th: an invoice for $5.73 was generated, but the associated payment was canceled within one second, and the invoice was subsequently voided.
The behavior you experienced does appear to be an edge case in how the billing system handled the canceled upgrade attempt. Even though the invoice was voided (meaning you don't owe anything), the system seems to have briefly flagged the account during that transition, which affected your deployments. You're right that this shouldn't have impacted services on an existing Hobby plan when the upgrade payment was simply canceled rather than failing on an outstanding balance.
Currently, your workspace shows 3 services running successfully (Orbitt Space Website, piersight-blog-notion, and Masonry Studio Website), 1 service in a failed state (umami in your Umami - Web Analytics project), and 17 services marked as removed across various projects. The umami failure may be related to this incident or could be because its supporting services (Postgres and Valkey) were removed.
For the services showing as removed, if it's been less than 30 days, you can redeploy them from the removed deployment's three-dot menu in each service. Your volume data is preserved—we don't delete volumes for billing-related stops.
I'll flag this internally for the team to review the billing logic around canceled upgrade attempts, since the invoice being voided should have prevented any service disruption. If you need any help getting specific services back online, let me know which ones and I can look into them further.
Status changed to Awaiting User Response Railway • about 2 months ago
2 months ago
The services running successfully are the ones I redeployed after noticing that all deployments were stopped. Two of them were critical marketing websites that shouldn't have been down.
Summarizing the issues:
1. The deployments were taken offline (your team will review billing logic)
2. No notification that they were taken offline (your team needs to look at this as well)
3. Project view indicated everything was online (your team needs to look at this as well)
If I hadn't checked in on Railway, I'd not have know unless someone told me the sites were down. Unfortunately my Uptime monitor was also on Railway and went offline with all the other services. That for me is very concerning that I had no way of knowing my deployments were taken offline.
Status changed to Awaiting Railway Response Railway • about 2 months ago
a month ago
Your three concerns are all valid and I appreciate you laying them out clearly.
On the notification gap—we don't currently send automatic email alerts when deployments are stopped. Deployment status notifications require setting up webhooks to Slack, Discord, or a custom endpoint. That's a reasonable expectation to have though, especially for billing-related service interruptions, and I understand the frustration given your uptime monitor was also hosted with us and went down simultaneously.
The Project view showing services as online when they weren't is definitely something that needs attention alongside the billing logic issue. Both have been flagged for the team to review.
For future monitoring resilience, you might consider an external uptime service (Uptime Robot, Better Stack, etc.) so you're not dependent on Railway being up to tell you Railway is down. Not ideal that it's necessary, but it would close that gap while the notification improvements are considered.
Status changed to Awaiting User Response Railway • about 2 months ago
a month ago
I agree that I should use an external uptime monitor to account for issues on my Railway account or Railway as a whole. Thanks for the suggestions on services to choose.
I'll setup the webhooks for notifications. Thanks for pointing it out.
Status changed to Awaiting Railway Response Railway • about 2 months ago
Status changed to Solved kosmischemusik • about 2 months ago
a month ago
Quick question. Which of these events would have alerted me when the deployments went offline?
Attachments
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
For the specific situation where deployments were stopped and marked as removed, the Deployment.removed event would be the one to watch for.
Status changed to Awaiting User Response Railway • about 1 month ago
a month ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • about 1 month ago