My Hobby service is stuck in an inconsistent deployment state.
stonedfury
HOBBYOP

2 days ago

My Hobby service is stuck in an inconsistent deployment state.

Project: LCARS-Prime

Service: stfc-assistant

Environment: production

Region: us-west2

Old deployment:

8bc1b684-05e1-40ef-8d17-e83947b07b63

The dashboard and API report SUCCESS with instance RUNNING, but Railway SSH reports:

"Your service's container is not running (status: stopping)."

New deployments build successfully but remain at Creating Containers / instance CREATED, produce no runtime logs, and eventually fail or stop.

The service has a persistent volume:

stfc-assistant-volume

Volume ID: 38491ae9-7f59-4b4c-a92a-bd727feed122

Mount path: /data

Size: 1 GB

State: READY

Please force-clear the stuck container/deployment state without deleting or altering the persistent volume. The volume contains the production SQLite database.

Awaiting User Response

14 Replies

Status changed to Awaiting Railway Response Railway 2 days ago


stonedfury
HOBBYOP

2 days ago

Please, when you resolve the issue, do not restart the application. I am currently running it locally and do not want them to cross.


stonedfury
HOBBYOP

2 days ago

Is there an ETA? Any response is better than no response.


2 days ago

What the deploy logs show when the container fails? Are there any error messages?


stonedfury
HOBBYOP

2 days ago

2026-07-25 08:30:22,728 | INFO | lcars_prime | ----------------------------------------

await self.coro(*args, **kwargs)

File "/app/cogs/moderation.py", line 939, in expire_temporary_punishments

records = await asyncio.to_thread(list_due_temporary_punishments)

          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

2026-07-25 09:19:22,756 | ERROR | discord.ext.tasks | Unhandled exception in internal background task 'expire_temporary_punishments'.

Traceback (most recent call last):

File "/app/services/moderation_service.py", line 367, in list_due_temporary_punishments

File "/app/.venv/lib/python3.13/site-packages/discord/ext/tasks/init.py", line 247, in _loop

    ^^^

...<6 lines>...

rows = connection.execute(

    (utc_now_iso(), max(1, min(limit, 500))),

    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

       ~~~~~~~~~~~~~~~~~~^

    """

File "/mise/installs/python/3.13.14/lib/python3.13/asyncio/threads.py", line 26, in to_thread

return await loop.run_in_executor(None, func_call)

       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/mise/installs/python/3.13.14/lib/python3.13/concurrent/futures/thread.py", line 59, in run

result = self.fn(*self.args, **self.kwargs)

).fetchall()

^

sqlite3.OperationalError: disk I/O error

2026-07-25 11:05:45,011 | INFO | discord.gateway | Shard ID None has successfully RESUMED session 862006b6bebb65fae99d78495d7e7d5f.

2026-07-25 14:40:10,978 | INFO | discord.gateway | Shard ID None has successfully RESUMED session 862006b6bebb65fae99d78495d7e7d5f.

I ran the SSH and it said it was stopped. I tried to remove it and it returns. All test show that there is an issue because I ran an update and it was interrupted because I ran out of space. This causes the two to conflict.


stonedfury
HOBBYOP

2 days ago

I have tried to stop it, and remove the build and redeploy an updated version. It will not let go of the session. So the update cannot go through. I updated to hobby and increased the disk size.

If you can get it to stop, release the session that would be great.


stonedfury
HOBBYOP

2 days ago

Please force-clear the stuck container/deployment state without deleting or altering the persistent volume. The volume contains the production SQLite database.


2 days ago

Redeploy wont delete files mounted on the volume, you can safely redeploy it, but if you really concern about the database file, you can manually download the files to your local device before redeploy


stonedfury
HOBBYOP

2 days ago

Did you read the initial post? The dashboard shows it deployed and working. It is not deployed or working. When i tried to stop it and remove it, it did not remove. Instead it went back and deployed. I tried to use SSH and it said it is stopped. However, the dashboard shows it active. It never let go of the sessions and therefore I cannot deploy an updated version. I tried to download the database to ensure it would be ok, and that is being blocked to because it is in use. When I stop and remove it still is in use. WHY? Because it has not let go of the session. Refer to the initial post which explains the problem. I have noticed that I am not the first to have the problem as there are other post within the last 14 days. I figured it would be an easy fix and I can go about my day. Is there no help that can see the system and do something?

I am stuck at this end in a loop.


stonedfury
HOBBYOP

2 days ago

Would you like me to record the issue. How I stop it, remove it, and wait for it to finish "removing" to which it never does and it redeploys? Then a video of the SSH showing it stopped while it shows in the dashboard as active? What proof do I need to provide to show that my service is not working? Aside from me telling you it is not working?


stonedfury
HOBBYOP

2 days ago

Thank you to whomever killed that session ID. Big thank you.


Status changed to Solved stonedfury 1 day ago


stonedfury
HOBBYOP

2 days ago

WoW, if you look I have 2 deployments. I guess you didn't kill that session.

broken.png

Attachments


Status changed to Awaiting Railway Response Railway 1 day ago


stonedfury
HOBBYOP

a day ago

I took screen shots of me removing, it removed and I will show you that it returns again. I can do a video. Why does it keep returning? Why won't it let the session go? Why hasn't a single person from railway looked at this and tried to fix it. Guess I will go back to x.com and any other ways of getting actual communication. Being your discord is a waste.

Screenshot 2026-07-26 110810.png

Screenshot 2026-07-26 110830.png

Screenshot 2026-07-26 110847.png


stonedfury

I took screen shots of me removing, it removed and I will show you that it returns again. I can do a video. Why does it keep returning? Why won't it let the session go? Why hasn't a single person from railway looked at this and tried to fix it. Guess I will go back to x.com and any other ways of getting actual communication. Being your discord is a waste. ![Screenshot 2026-07-26 110810.png](https://station-server.railway.com/attachments/att_01kyfsx1w5fzstfee5svacp1pn) ![Screenshot 2026-07-26 110830.png](https://station-server.railway.com/attachments/att_01kyfsy9kqf248pj9fbm7xk6nk) ![Screenshot 2026-07-26 110847.png](https://station-server.railway.com/attachments/att_01kyfsxne2fansvkx749vtavt2)

stonedfury
HOBBYOP

a day ago

It's back. I showed you a screen shot of me removing it. It's back!


2 hours ago

Thanks for your patience here, and sorry for the runaround. You were right that something was stuck on our side. The container for your old July 25 deployment had already been stopped on the underlying host, but our database still recorded it as running. Every removal attempt then failed against that missing container and rolled the deployment back to an active state, which is why it kept reappearing no matter how many times you removed it.

We have now cleared that stuck record manually. Your volume was not touched, so the SQLite database on it is intact.

One thing to flag: the deployment you triggered this morning did complete successfully and appears to be running now. Since you mentioned you are running the bot locally and do not want two instances live, you may want to remove that newest deployment from the dashboard yourself. Removals should go through normally now that the stuck record is gone.


Status changed to Awaiting User Response Railway about 2 hours ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...