a month ago
Project: diplomatic-vision (project ID 3ea8dc4b-86f4-4c39-9124-692be126bcb1)
Environment: production (f2604ad9-ea4d-4d48-a93b-239e4a4fc794)
We had a service called "function-bun" in this project that we deleted on 2026-07-14. Instead of terminating its running container, the delete appears to have orphaned it. It's still alive and still connected to our Postgres database today (2026-07-21), a week after deletion and nearly a month after it was last deployed.
Evidence from pg_stat_activity on our Postgres instance:
pid=70826 backend_start=2026-06-23 15:22:15 UTC client_addr=fd12:8b:4824:1:3000:31:dbb9:2fad
pid=70840 backend_start=2026-06-23 15:29:00 UTC client_addr=fd12:8b:4824:1:3000:31:dbb9:2fad
(10 total connections from that host)
pid=70895 backend_start=2026-06-23 16:04:44 UTC client_addr=fd12:8b:4824:1:3000:42:e8d7:c61e
Both hosts have a backend_start of 2026-06-23, before our last ~90 backend deploys, and their client_addr is an internal Railway IPv6 (fd12::/48), i.e. a container inside this project, not an external client. Our actual backend service (stocktrading-backend) redeploys cleanly on every push and its containers all show recent backend_start times; these two do not correspond to any currently-listed service or deployment.
Impact: this orphaned process is still running months-old application code with an in-process scheduler, and is actively writing incorrect data to our production database on a daily schedule, because its code predates several bug fixes we've since shipped to main. We have no lever on our side to stop it since it doesn't show up as a service or deployment anymore.
Could you please locate and terminate the containers at fd12:8b:4824:1:3000:31:dbb9:2fad and fd12:8b:4824:1:3000:42:e8d7:c61e (both originally provisioned as the "function-bun" service, deleted 2026-07-14)?
We're also rotating our Postgres password as a belt-and-suspenders fix on our end, but would still like these orphaned containers reaped so this doesn't recur with the next deleted/replaced service. Happy to provide more pg_stat_activity output or deployment logs if useful.
1 Replies
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
We found them and you were right about the symptom. Two containers from your stocktrading-backend service got stuck during removal on June 23 and kept running old code ever since, which matches the connection times you pulled from pg_stat_activity. The function-bun deletion turned out to be unrelated.
We've stopped both containers, so the stale scheduler writes will stop immediately. Rotating your Postgres password is still a sensible precaution, but nothing else is connected from our side now.
Since those two containers were billed as extra usage while they were stuck, we've also added a $5 credit to your account to cover it.
This was a bug in our deployment teardown, not anything you did. Thanks for the unusually precise report, it made this quick to track down.
Railway Team
Status changed to Awaiting User Response Railway • about 1 month ago
23 days ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • 23 days ago