16 days ago
Service: backend
Deployment ID: 7800adfe-da70-4236-a61c-01f18ae730cd
Error: no space left on device on BuildKit metadata database
Note: Four consecutive deployments failing with identical error
7 Replies
Status changed to Awaiting Railway Response Railway • 16 days ago
16 days ago
The "no space left on device" error here is fallout from yesterday's incident, where GCP brought our builders back up in an incorrect state. That's been fixed on our end, so canceling and retrying the failed builds should get you unblocked. Sorry for the disruption this caused.
Status changed to Awaiting User Response Railway • 16 days ago
16 days ago
Hey there,
We had a builder disk pressure issue across our V3 fleet (us-west1 region) following the post-outage recovery on May 20. The build queue is fully healthy again now; end-to-end deploys across the platform are completing in under a minute.
I just kicked a fresh redeploy on your service from my side. It should land on a clean builder.
If you're still hitting the same "no space left on device" error after this, reply with the latest deployment ID and I'll route it for direct attention.
Sorry for the disruption,
Angelo
angelo-railway
Hey there, We had a builder disk pressure issue across our V3 fleet (us-west1 region) following the post-outage recovery on May 20. The build queue is fully healthy again now; end-to-end deploys across the platform are completing in under a minute. I just kicked a fresh redeploy on your service from my side. It should land on a clean builder. If you're still hitting the same "no space left on device" error after this, reply with the latest deployment ID and I'll route it for direct attention. Sorry for the disruption, Angelo
16 days ago
Hi, production service down all afternoon since the GCP outage. Persistent SqliteError: attempt to write a readonly database (code: SQLITE_READONLY) crash loop. Tried everything from the dashboard: At first it said it was deployed but Post deploy never did anything.
Restart — same error
Redeployed older known-good commits — same error
Restored from a 9-hour-old backup (new volume created from backup) — same error on the brand-new volume
The fact that a freshly restored volume is also mounting read-only strongly suggests the underlying node is unhealthy. Per your FAQ, please move my volume to a healthy node or let me know if this is just because the system is down.
⚠️ STRICT SCOPE OF WORK — PLEASE READ CAREFULLY:
The ONLY action I'm requesting is: move the existing /data volume — with all its data intact — to a healthy node so it mounts read-write again.
DO NOT touch, modify, or change anything else. Specifically:
DATA & STORAGE:
DO NOT delete the volume
DO NOT delete, modify, or expire any backups (I have a daily snapshot from 9 hours ago and a manual snapshot from 19 hours ago — both must remain)
DO NOT reset, wipe, recreate, or reinitialize the database
DO NOT alter the contents of the /data volume in any way
CODE & DEPLOYMENT:
DO NOT modify, rebuild, or redeploy from any specific commit
DO NOT change the Dockerfile, build settings, or start command
DO NOT touch the GitHub integration or auto-deploy settings
DO NOT modify any source code (it's in my GitHub repo — I assume you cannot anyway, just confirming)
SERVICE CONFIGURATION:
DO NOT modify environment variables
DO NOT change the domain, custom domain settings, or DNS routing
DO NOT change networking, ports, or replicas
DO NOT modify the service, project, or environment settings
DO NOT delete or rebuild the service itself
If something is unclear, please reply in this thread and ask me before taking any action. I'd rather wait longer for a careful fix than have anything else broken.
Context: I run a real estate brokerage on this platform (NexPath). Every record — agents, transactions, revshare ledgers, CDAs, broker reports, documents — is critical business data. The system is otherwise working perfectly; this is purely a read-only volume mount issue it looks like from the GCP outage.
IDs:
Project ID: 42de96c2-5bf6-425c-bc70-6c5135f7c340
Service ID: d0f2734e-239b-47bf-9123-9237dbb387d3
Environment ID: d6c560ca-24c9-4a35-866d-6b5cf32e7be0
Service name: web
Domain: nexpathplatform.com / www.nexpathplatform.com
Mount path: /data
Thanks for your help.
Status changed to Awaiting Railway Response Railway • 16 days ago
bmclain07
Hi, production service down all afternoon since the GCP outage. Persistent SqliteError: attempt to write a readonly database (code: SQLITE_READONLY) crash loop. Tried everything from the dashboard: At first it said it was deployed but Post deploy never did anything. Restart — same error Redeployed older known-good commits — same error Restored from a 9-hour-old backup (new volume created from backup) — same error on the brand-new volume The fact that a freshly restored volume is also mounting read-only strongly suggests the underlying node is unhealthy. Per your FAQ, please move my volume to a healthy node or let me know if this is just because the system is down. ⚠️ STRICT SCOPE OF WORK — PLEASE READ CAREFULLY: The ONLY action I'm requesting is: move the existing /data volume — with all its data intact — to a healthy node so it mounts read-write again. DO NOT touch, modify, or change anything else. Specifically: DATA & STORAGE: DO NOT delete the volume DO NOT delete, modify, or expire any backups (I have a daily snapshot from 9 hours ago and a manual snapshot from 19 hours ago — both must remain) DO NOT reset, wipe, recreate, or reinitialize the database DO NOT alter the contents of the /data volume in any way CODE & DEPLOYMENT: DO NOT modify, rebuild, or redeploy from any specific commit DO NOT change the Dockerfile, build settings, or start command DO NOT touch the GitHub integration or auto-deploy settings DO NOT modify any source code (it's in my GitHub repo — I assume you cannot anyway, just confirming) SERVICE CONFIGURATION: DO NOT modify environment variables DO NOT change the domain, custom domain settings, or DNS routing DO NOT change networking, ports, or replicas DO NOT modify the service, project, or environment settings DO NOT delete or rebuild the service itself If something is unclear, please reply in this thread and ask me before taking any action. I'd rather wait longer for a careful fix than have anything else broken. Context: I run a real estate brokerage on this platform (NexPath). Every record — agents, transactions, revshare ledgers, CDAs, broker reports, documents — is critical business data. The system is otherwise working perfectly; this is purely a read-only volume mount issue it looks like from the GCP outage. IDs: Project ID: 42de96c2-5bf6-425c-bc70-6c5135f7c340 Service ID: d0f2734e-239b-47bf-9123-9237dbb387d3 Environment ID: d6c560ca-24c9-4a35-866d-6b5cf32e7be0 Service name: web Domain: nexpathplatform.com / www.nexpathplatform.com Mount path: /data Thanks for your help.
16 days ago
@angelo can u check my account and see what needs to take place for my site to be back up? I don't need anything deleted or any data lost. Im a brokerage and need everything there
16 days ago
Hey @bmclain07, please keep your issue on your own thread rather than posting in another customer's: https://station.railway.com/questions/i-need-help-ba9d63e9
Status changed to Awaiting User Response Railway • 16 days ago
brody
Hey @bmclain07, please keep your issue on your own thread rather than posting in another customer's: https://station.railway.com/questions/i-need-help-ba9d63e9
16 days ago
Yes sir! I have got my stuff taken care of. sorry i didn't realize. Disregard anything from me
Status changed to Awaiting Railway Response Railway • 16 days ago
bmclain07
@angelo can u check my account and see what needs to take place for my site to be back up? I don't need anything deleted or any data lost. Im a brokerage and need everything there
16 days ago
I got my stuff taken care off! Thank you, disregard my request! Thank yall
Status changed to Solved Railway • 16 days ago