15 days ago
Every hour at ~:01–:03 UTC, ALL service deployments in my prod environment are
removed within ~15 seconds — including my Postgres database — while in SUCCESS
state. Nothing replaces them, so the environment stays down until I manually
redeploy, and then it is wiped again at the next hour.
Project: nevo — 6fdbd117-676e-49c1-9454-883c7a0973db
Environment: prod — 28aa9fbb-699c-4edc-adfb-d633e72f7659
Confirmed waves (UTC), from the events API:
Aug 6: 07:01 · 08:01 (9 services) · 09:01 (8 services) · 10:03
Aug 5: 11:03 · 12:02 · 14:03 · 16:02 (8 services each)
After each wave the API domain returns 404 "Application not found" (no container),
and Postgres logs show a clean SIGTERM shutdown — checkpoint completes, then
"Stopping Container". No error precedes it.
Already ruled out:
-
Cron schedules: all services now have cronSchedule = null. The Aug 6 10:03
wave happened with zero crons configured anywhere.
-
Usage limits: state = ACTIVE, currentUsage 14.03 of hardLimit 20,
isOverLimit = false.
-
App sleeping: sleepApplication = false.
-
My automation: no project tokens, no other members, no webhooks.
-
User action: the audit log records my own deploys, but has NO entry for any
of these removals — they appear only in the events API, with no actor.
-
Supersession: no new deployment is created around the waves.
One cron service was oom_killed once (Aug 6, 09:19) and I'm moving that job off
Railway — but the wipes also occur when it isn't running, and my database peaks
at 0.34 GB against an 8 GB limit.
Questions: what is removing these deployments and why is there no audit entry?
Is this platform behaviour, account enforcement, or something in my config?
Would recreating the environment avoid it?
1 Replies
15 days ago
Your workspace's compute hard limit ($20) was crossed on Aug 4 at 20:03 UTC, which is what triggered the deployment removals. When the hard limit is reached, all workloads are taken offline to stop further usage, and manually redeploying without first raising or removing the limit will result in those deployments being taken down again. To restore your services immediately, raise or remove the hard limit at Workspace Usage, and your billing cycle resets on Aug 7 which will also clear the condition.
Status changed to Awaiting User Response Railway • 15 days ago
Status changed to Solved enekoruizdeloizaga • 15 days ago