All deployments in one environment are removed every hour (~:01 UTC), including Postgres
enekoruizdeloizaga
HOBBYOP

15 days ago

Every hour at ~:01–:03 UTC, ALL service deployments in my prod environment are

removed within ~15 seconds — including my Postgres database — while in SUCCESS

state. Nothing replaces them, so the environment stays down until I manually

redeploy, and then it is wiped again at the next hour.

Project: nevo — 6fdbd117-676e-49c1-9454-883c7a0973db

Environment: prod — 28aa9fbb-699c-4edc-adfb-d633e72f7659

Confirmed waves (UTC), from the events API:

Aug 6: 07:01 · 08:01 (9 services) · 09:01 (8 services) · 10:03

Aug 5: 11:03 · 12:02 · 14:03 · 16:02 (8 services each)

After each wave the API domain returns 404 "Application not found" (no container),

and Postgres logs show a clean SIGTERM shutdown — checkpoint completes, then

"Stopping Container". No error precedes it.

Already ruled out:

  • Cron schedules: all services now have cronSchedule = null. The Aug 6 10:03

    wave happened with zero crons configured anywhere.

  • Usage limits: state = ACTIVE, currentUsage 14.03 of hardLimit 20,

    isOverLimit = false.

  • App sleeping: sleepApplication = false.

  • My automation: no project tokens, no other members, no webhooks.

  • User action: the audit log records my own deploys, but has NO entry for any

    of these removals — they appear only in the events API, with no actor.

  • Supersession: no new deployment is created around the waves.

One cron service was oom_killed once (Aug 6, 09:19) and I'm moving that job off

Railway — but the wipes also occur when it isn't running, and my database peaks

at 0.34 GB against an 8 GB limit.

Questions: what is removing these deployments and why is there no audit entry?

Is this platform behaviour, account enforcement, or something in my config?

Would recreating the environment avoid it?

Solved

1 Replies

Railway
BOT

15 days ago

Your workspace's compute hard limit ($20) was crossed on Aug 4 at 20:03 UTC, which is what triggered the deployment removals. When the hard limit is reached, all workloads are taken offline to stop further usage, and manually redeploying without first raising or removing the limit will result in those deployments being taken down again. To restore your services immediately, raise or remove the hard limit at Workspace Usage, and your billing cycle resets on Aug 7 which will also clear the condition.


Status changed to Awaiting User Response Railway 15 days ago


Status changed to Solved enekoruizdeloizaga 15 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...