Four consecutive deployments failing with identical error
scarlet-artisan
PROOP

a month ago

Service: backend

Deployment ID: 7800adfe-da70-4236-a61c-01f18ae730cd

Error: no space left on device on BuildKit metadata database

Note: Four consecutive deployments failing with identical error

Solved

7 Replies

Status changed to Awaiting Railway Response Railway about 1 month ago


a month ago

The "no space left on device" error here is fallout from yesterday's incident, where GCP brought our builders back up in an incorrect state. That's been fixed on our end, so canceling and retrying the failed builds should get you unblocked. Sorry for the disruption this caused.


Status changed to Awaiting User Response Railway about 1 month ago


Hey there,

We had a builder disk pressure issue across our V3 fleet (us-west1 region) following the post-outage recovery on May 20. The build queue is fully healthy again now; end-to-end deploys across the platform are completing in under a minute.

I just kicked a fresh redeploy on your service from my side. It should land on a clean builder.

If you're still hitting the same "no space left on device" error after this, reply with the latest deployment ID and I'll route it for direct attention.

Sorry for the disruption,

Angelo


angelo-railway

Hey there, We had a builder disk pressure issue across our V3 fleet (us-west1 region) following the post-outage recovery on May 20. The build queue is fully healthy again now; end-to-end deploys across the platform are completing in under a minute. I just kicked a fresh redeploy on your service from my side. It should land on a clean builder. If you're still hitting the same "no space left on device" error after this, reply with the latest deployment ID and I'll route it for direct attention. Sorry for the disruption, Angelo

bmclain07
PRO

a month ago

Hi, production service down all afternoon since the GCP outage. Persistent SqliteError: attempt to write a readonly database (code: SQLITE_READONLY) crash loop. Tried everything from the dashboard: At first it said it was deployed but Post deploy never did anything.

Restart — same error

Redeployed older known-good commits — same error

Restored from a 9-hour-old backup (new volume created from backup) — same error on the brand-new volume

The fact that a freshly restored volume is also mounting read-only strongly suggests the underlying node is unhealthy. Per your FAQ, please move my volume to a healthy node or let me know if this is just because the system is down.

⚠️ STRICT SCOPE OF WORK — PLEASE READ CAREFULLY:

The ONLY action I'm requesting is: move the existing /data volume — with all its data intact — to a healthy node so it mounts read-write again.

DO NOT touch, modify, or change anything else. Specifically:

DATA & STORAGE:

DO NOT delete the volume

DO NOT delete, modify, or expire any backups (I have a daily snapshot from 9 hours ago and a manual snapshot from 19 hours ago — both must remain)

DO NOT reset, wipe, recreate, or reinitialize the database

DO NOT alter the contents of the /data volume in any way

CODE & DEPLOYMENT:

DO NOT modify, rebuild, or redeploy from any specific commit

DO NOT change the Dockerfile, build settings, or start command

DO NOT touch the GitHub integration or auto-deploy settings

DO NOT modify any source code (it's in my GitHub repo — I assume you cannot anyway, just confirming)

SERVICE CONFIGURATION:

DO NOT modify environment variables

DO NOT change the domain, custom domain settings, or DNS routing

DO NOT change networking, ports, or replicas

DO NOT modify the service, project, or environment settings

DO NOT delete or rebuild the service itself

If something is unclear, please reply in this thread and ask me before taking any action. I'd rather wait longer for a careful fix than have anything else broken.

Context: I run a real estate brokerage on this platform (NexPath). Every record — agents, transactions, revshare ledgers, CDAs, broker reports, documents — is critical business data. The system is otherwise working perfectly; this is purely a read-only volume mount issue it looks like from the GCP outage.

IDs:

Project ID: 42de96c2-5bf6-425c-bc70-6c5135f7c340

Service ID: d0f2734e-239b-47bf-9123-9237dbb387d3

Environment ID: d6c560ca-24c9-4a35-866d-6b5cf32e7be0

Service name: web

Domain: nexpathplatform.com / www.nexpathplatform.com

Mount path: /data

Thanks for your help.


Status changed to Awaiting Railway Response Railway about 1 month ago


bmclain07

Hi, production service down all afternoon since the GCP outage. Persistent SqliteError: attempt to write a readonly database (code: SQLITE_READONLY) crash loop. Tried everything from the dashboard: At first it said it was deployed but Post deploy never did anything. Restart — same error Redeployed older known-good commits — same error Restored from a 9-hour-old backup (new volume created from backup) — same error on the brand-new volume The fact that a freshly restored volume is also mounting read-only strongly suggests the underlying node is unhealthy. Per your FAQ, please move my volume to a healthy node or let me know if this is just because the system is down. ⚠️ STRICT SCOPE OF WORK — PLEASE READ CAREFULLY: The ONLY action I'm requesting is: move the existing /data volume — with all its data intact — to a healthy node so it mounts read-write again. DO NOT touch, modify, or change anything else. Specifically: DATA & STORAGE: DO NOT delete the volume DO NOT delete, modify, or expire any backups (I have a daily snapshot from 9 hours ago and a manual snapshot from 19 hours ago — both must remain) DO NOT reset, wipe, recreate, or reinitialize the database DO NOT alter the contents of the /data volume in any way CODE & DEPLOYMENT: DO NOT modify, rebuild, or redeploy from any specific commit DO NOT change the Dockerfile, build settings, or start command DO NOT touch the GitHub integration or auto-deploy settings DO NOT modify any source code (it's in my GitHub repo — I assume you cannot anyway, just confirming) SERVICE CONFIGURATION: DO NOT modify environment variables DO NOT change the domain, custom domain settings, or DNS routing DO NOT change networking, ports, or replicas DO NOT modify the service, project, or environment settings DO NOT delete or rebuild the service itself If something is unclear, please reply in this thread and ask me before taking any action. I'd rather wait longer for a careful fix than have anything else broken. Context: I run a real estate brokerage on this platform (NexPath). Every record — agents, transactions, revshare ledgers, CDAs, broker reports, documents — is critical business data. The system is otherwise working perfectly; this is purely a read-only volume mount issue it looks like from the GCP outage. IDs: Project ID: 42de96c2-5bf6-425c-bc70-6c5135f7c340 Service ID: d0f2734e-239b-47bf-9123-9237dbb387d3 Environment ID: d6c560ca-24c9-4a35-866d-6b5cf32e7be0 Service name: web Domain: nexpathplatform.com / www.nexpathplatform.com Mount path: /data Thanks for your help.

bmclain07
PRO

a month ago

@angelo can u check my account and see what needs to take place for my site to be back up? I don't need anything deleted or any data lost. Im a brokerage and need everything there


a month ago

Hey @bmclain07, please keep your issue on your own thread rather than posting in another customer's: https://station.railway.com/questions/i-need-help-ba9d63e9


Status changed to Awaiting User Response Railway about 1 month ago


brody

Hey @bmclain07, please keep your issue on your own thread rather than posting in another customer's: https://station.railway.com/questions/i-need-help-ba9d63e9

bmclain07
PRO

a month ago

Yes sir! I have got my stuff taken care of. sorry i didn't realize. Disregard anything from me


Status changed to Awaiting Railway Response Railway about 1 month ago


bmclain07

@angelo can u check my account and see what needs to take place for my site to be back up? I don't need anything deleted or any data lost. Im a brokerage and need everything there

bmclain07
PRO

a month ago

I got my stuff taken care off! Thank you, disregard my request! Thank yall


Status changed to Solved Railway about 1 month ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...