Hello,

Admittedly, we get many tickets asking us how to bring back up services that have been shut down, and in the end, we have never seen any reports such as yours, so the exact steps on how you went about bringing your services back online would be most helpful!

Best,
Brody

Oh, interesting! Yeah, so here's the chain of events as exactly as I can recall it:

-Receive overnight email that usage limit has been exceeded.
-Next morning, go into settings and update usage limit

-Check app, still down

-Check production instance. Databases (Postgres, Redis) are not deployed but no errors. Services (Django, Celery, Celery Beat) are down with red warning text that says something like "Build failed."

-Select "Redeploy" for all apps

-When prompted with "This build failed, still redeploy?" click yes

-Switch to staging

-Staging services are also down, but just suspended. No red warning text.

-Redeploy those services, too

-At this point, notice that databases are down

-Redeploy those too, on both instances

-All services queued or building

-Wait for builds to complete, no issues

-Navigate to prod app URL

-Sign in, start testing current feature in development

-Wait, it shouldn't work like that

-Check git code

-Looks fine

-Re-check Railway

-Current build is "Merge pull request #63" but services are deployed from "Merge pull request #24"

-Scroll through deployment history

-Pull request #24 was last pull request that failed to build

-Redeploy services from PR #63

-Prod now works fine, no issue

-Staging also fine, no issue, worked as expected all along

So my only guess is that exceeding the usage limit caused the build to fail in some fashion and when Railway re-loaded the build after failure it chose the last failed build, which was an older one? It didn't change the build for staging, but it's possible that's because there are no failed builds in my staging deployment history.

Hope that helps!

Hello,

Thank you for the detailed log, I think I see what went wrong.

> So my only guess is that exceeding the usage limit caused the build to fail in some fashion

Not quite, we simply remove the active deployment, it then gets placed in your history for you to redeploy at your leisure.

> when Railway re-loaded the build after failure it chose the last failed build

Hmmm, by the sounds of it, you chose the last failed build... "When prompted with "This build failed, still redeploy?" click yes"

Sure the failed build appeared in the proment location, but the deploy you wanted was in your history.

Regardless, we know this experience is not good to say the least, and thus, we have a project that when shipped will simply allow users to deploy the removed deployments with a few clicks after you restore billing / increase your limit / load more credits / etc.

Ok! Whatever UI you feel is appropriate, I just figured I'd let you all know.