My Hobby service is stuck in an inconsistent deployment state.
stonedfury
HOBBYOP

2 months ago

My Hobby service is stuck in an inconsistent deployment state.

Project: LCARS-Prime

Service: stfc-assistant

Environment: production

Region: us-west2

Old deployment:

8bc1b684-05e1-40ef-8d17-e83947b07b63

The dashboard and API report SUCCESS with instance RUNNING, but Railway SSH reports:

"Your service's container is not running (status: stopping)."

New deployments build successfully but remain at Creating Containers / instance CREATED, produce no runtime logs, and eventually fail or stop.

The service has a persistent volume:

stfc-assistant-volume

Volume ID: 38491ae9-7f59-4b4c-a92a-bd727feed122

Mount path: /data

Size: 1 GB

State: READY

Please force-clear the stuck container/deployment state without deleting or altering the persistent volume. The volume contains the production SQLite database.

Solved

15 Replies

Status changed to Awaiting Railway Response Railway • 2 months ago


stonedfury
HOBBYOP

2 months ago

Please, when you resolve the issue, do not restart the application. I am currently running it locally and do not want them to cross.


stonedfury
HOBBYOP

2 months ago

Is there an ETA? Any response is better than no response.


2 months ago

What the deploy logs show when the container fails? Are there any error messages?


stonedfury
HOBBYOP

2 months ago

2026-07-25 08:30:22,728 | INFO | lcars_prime | ----------------------------------------

await self.coro(*args, **kwargs)

File "/app/cogs/moderation.py", line 939, in expire_temporary_punishments

records = await asyncio.to_thread(list_due_temporary_punishments)

          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

2026-07-25 09:19:22,756 | ERROR | discord.ext.tasks | Unhandled exception in internal background task 'expire_temporary_punishments'.

Traceback (most recent call last):

File "/app/services/moderation_service.py", line 367, in list_due_temporary_punishments

File "/app/.venv/lib/python3.13/site-packages/discord/ext/tasks/init.py", line 247, in _loop

    ^^^

...<6 lines>...

rows = connection.execute(

    (utc_now_iso(), max(1, min(limit, 500))),

    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

       ~~~~~~~~~~~~~~~~~~^

    """

File "/mise/installs/python/3.13.14/lib/python3.13/asyncio/threads.py", line 26, in to_thread

return await loop.run_in_executor(None, func_call)

       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/mise/installs/python/3.13.14/lib/python3.13/concurrent/futures/thread.py", line 59, in run

result = self.fn(*self.args, **self.kwargs)

).fetchall()

^

sqlite3.OperationalError: disk I/O error

2026-07-25 11:05:45,011 | INFO | discord.gateway | Shard ID None has successfully RESUMED session 862006b6bebb65fae99d78495d7e7d5f.

2026-07-25 14:40:10,978 | INFO | discord.gateway | Shard ID None has successfully RESUMED session 862006b6bebb65fae99d78495d7e7d5f.

I ran the SSH and it said it was stopped. I tried to remove it and it returns. All test show that there is an issue because I ran an update and it was interrupted because I ran out of space. This causes the two to conflict.


stonedfury
HOBBYOP

2 months ago

I have tried to stop it, and remove the build and redeploy an updated version. It will not let go of the session. So the update cannot go through. I updated to hobby and increased the disk size.

If you can get it to stop, release the session that would be great.


stonedfury
HOBBYOP

2 months ago

Please force-clear the stuck container/deployment state without deleting or altering the persistent volume. The volume contains the production SQLite database.


2 months ago

Redeploy wont delete files mounted on the volume, you can safely redeploy it, but if you really concern about the database file, you can manually download the files to your local device before redeploy


stonedfury
HOBBYOP

2 months ago

Did you read the initial post? The dashboard shows it deployed and working. It is not deployed or working. When i tried to stop it and remove it, it did not remove. Instead it went back and deployed. I tried to use SSH and it said it is stopped. However, the dashboard shows it active. It never let go of the sessions and therefore I cannot deploy an updated version. I tried to download the database to ensure it would be ok, and that is being blocked to because it is in use. When I stop and remove it still is in use. WHY? Because it has not let go of the session. Refer to the initial post which explains the problem. I have noticed that I am not the first to have the problem as there are other post within the last 14 days. I figured it would be an easy fix and I can go about my day. Is there no help that can see the system and do something?

I am stuck at this end in a loop.


stonedfury
HOBBYOP

2 months ago

Would you like me to record the issue. How I stop it, remove it, and wait for it to finish "removing" to which it never does and it redeploys? Then a video of the SSH showing it stopped while it shows in the dashboard as active? What proof do I need to provide to show that my service is not working? Aside from me telling you it is not working?


stonedfury
HOBBYOP

2 months ago

Thank you to whomever killed that session ID. Big thank you.


Status changed to Solved stonedfury • 2 months ago


stonedfury
HOBBYOP

2 months ago

WoW, if you look I have 2 deployments. I guess you didn't kill that session.

broken.png

Attachments


Status changed to Awaiting Railway Response Railway • 2 months ago


stonedfury
HOBBYOP

2 months ago

I took screen shots of me removing, it removed and I will show you that it returns again. I can do a video. Why does it keep returning? Why won't it let the session go? Why hasn't a single person from railway looked at this and tried to fix it. Guess I will go back to x.com and any other ways of getting actual communication. Being your discord is a waste.

Screenshot 2026-07-26 110810.png

Screenshot 2026-07-26 110830.png

Screenshot 2026-07-26 110847.png


stonedfury

I took screen shots of me removing, it removed and I will show you that it returns again. I can do a video. Why does it keep returning? Why won't it let the session go? Why hasn't a single person from railway looked at this and tried to fix it. Guess I will go back to x.com and any other ways of getting actual communication. Being your discord is a waste. ![Screenshot 2026-07-26 110810.png](https://station-server.railway.com/attachments/att_01kyfsx1w5fzstfee5svacp1pn) ![Screenshot 2026-07-26 110830.png](https://station-server.railway.com/attachments/att_01kyfsy9kqf248pj9fbm7xk6nk) ![Screenshot 2026-07-26 110847.png](https://station-server.railway.com/attachments/att_01kyfsxne2fansvkx749vtavt2)

stonedfury
HOBBYOP

2 months ago

It's back. I showed you a screen shot of me removing it. It's back!


2 months ago

Thanks for your patience here, and sorry for the runaround. You were right that something was stuck on our side. The container for your old July 25 deployment had already been stopped on the underlying host, but our database still recorded it as running. Every removal attempt then failed against that missing container and rolled the deployment back to an active state, which is why it kept reappearing no matter how many times you removed it.

We have now cleared that stuck record manually. Your volume was not touched, so the SQLite database on it is intact.

One thing to flag: the deployment you triggered this morning did complete successfully and appears to be running now. Since you mentioned you are running the bot locally and do not want two instances live, you may want to remove that newest deployment from the dashboard yourself. Removals should go through normally now that the stuck record is gone.


Status changed to Awaiting User Response Railway • 2 months ago


nico

Thanks for your patience here, and sorry for the runaround. You were right that something was stuck on our side. The container for your old July 25 deployment had already been stopped on the underlying host, but our database still recorded it as running. Every removal attempt then failed against that missing container and rolled the deployment back to an active state, which is why it kept reappearing no matter how many times you removed it. We have now cleared that stuck record manually. Your volume was not touched, so the SQLite database on it is intact. One thing to flag: the deployment you triggered this morning did complete successfully and appears to be running now. Since you mentioned you are running the bot locally and do not want two instances live, you may want to remove that newest deployment from the dashboard yourself. Removals should go through normally now that the stuck record is gone.

stonedfury
HOBBYOP

2 months ago

Thank you. I saw that it was gone earlier. I am very grateful that it has been delt with.


Status changed to Awaiting Railway Response Railway • 2 months ago


Status changed to Solved stonedfury • 2 months ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...