my error isn't any of these, i am getting an application failed to respond( 502 Bad Gateway), i have a staging and production environment and the staging is up and running as expected, production is not . I redeployed, restarted and still getting the same error. i have cleared my cache and did a hard refresh. No errors in my deploy logs and my git hub can still push out changes . Any other things i can check or just wait for the all clear status from you guys ?

Postgres crash loop here, plus a more concerning issue beyond the standard scenario.

Project ID: 2497f3fe-fd67-4c1f-b8fb-c872f62c98f9

Service ID: pgvector (service in project trustworthy-forgiveness)

Volume currently attached: vol_jwwp0ua41m4nkdce

Standard crash loop symptoms (initdb against non-empty PGDATA on every restart), but diagnostic SSH into the volume shows the data on disk is from this service's initial provisioning on 2026-02-23 — three months of production writes are missing. The 97 MB backup may or may not contain my real data; the dashboard restore option only lets me restore in-place to the broken service, not to a fresh one for safe inspection.

Full technical detail and diagnostic findings in my existing thread:

https://station.railway.com/questions/postgres-crash-loop-2338ad96

Volume relocation as you mentioned would help with the crash loop, but I also need to know whether you have access to a snapshot or recovery state of my actual production data from before the incident. Happy to provide anything else useful.

My services were restored, but no subsequent deployments are working essentially freezing the state of all services. I have multiple services stuck in Initializing for the past several hours....

Vocês se precaveram de não reembolsar prejuízos catastróficos, mas não se precaveram de um plano B para isso? lamentável a confiança esta sendo perdida.

▎ Matches the Postgres catatonit: failed to exec pid1 case from your FAQ.

▎

▎ Project ID: b9568529-d4e1-4c85-8260-953a98c2b343

▎ Service ID: 1bd97dc8-17db-4452-b08f-d10cc5644edf (Postgres)

▎ Environment ID: 470a95a9-4e24-438f-b5e0-869887e92c6d ▎

▎ Crash-looping since the outage window (~02:25 UTC May 20). Tried multiple redeploys via dashboard, no recovery — service immediately exits with catatonit: failed to exec pid1: No such file or directory after each volume mount. ▎

▎ Per your FAQ this likely needs the volume moved to a healthy node. Volume contains live customer data — please don't reprovision empty.

▎

▎ Thanks.

Postgres crash loop here, plus a more concerning issue beyond the standard scenario.

Project ID: 44956dd6-9463-4d94-afcf-d77c03ebfda8

Service ID: c580447b-4060-4931-9b3a-6a2a69d9dfac

Volume currently attached: 96104140-7ed8-488f-ae22-551d385cf2b2

I'm having the same problem. I have a production application and my clients can't access it; it returns a 502 Bad Gateway error. I don't know what to do.

I still can't deploy it, even though I want to do a new build. I see these error texts: 1.) Builds are slow to progress We have pushed a fix and are now monitoring the incident.

2.) Limited Access

Deploys have been paused temporarily

Im in the same boat after 12 hours now

Project ID: 7e7dcdb2-7e66-46b4-81fc-26369e8e307d

Service ID: 10175118-324e-444b-aa2e-931963e030ba

Limited Access: Deploys have been paused temporarily

I have hobby plan and I got this "Deploys have been paused temporarily" why?

this is my Project ID: 41c4676a-c09e-4278-bdd2-3d5a6f23add0

I also have this problem. When can I expect to be able to deploy again?

If I upgrade to PRO will I be able to deploy new builds? We have a critical need and are considering upgrading, but not sure if it's worth it if PRO level will not let us deploy new builds. Do we know when hobby new builds will be allowed?

Same symptoms here on Pro — web service: project c6772164-8ef4-410a-812b-0cd527142449, service 6b14047b-6abc-47d1-9b87-73b6bb60b7d1. Postgres + Redis are healthy and reachable, deploys show Active/successful, but every request returns 502 in <20ms with X-Railway-Fallback: true. Tried redeploys, restarts, multiple startCommand variations — no luck. Private thread already filed with full detail. Thanks.

Wow, you have more than 12 hours with this issue? really? Im blocked on new deployments.

My docker images have not been built and throw errors, as an exampl

Build Failed: build daemon returned an error < failed to solve: ResourceExhausted: failed to read dockerfile: failed to prepare as wgsgybnscmz48o1qe57ifegdd: failed to create prepare snapshot dir: failed to create temp dir: mkdir /var/lib/buildkit/runc-overlayfs/snapshots/snapshots/new-752968274: no space left on device >

My Postgres is stuck in a crash loop with catatonit: failed to exec pid1. Tried multiple redeploys, no recovery.

Project ID: 0ae91dc6-42eb-4039-acdf-0240d1720849

is this because of the issues ? internal

load build definition from backend/Dockerfile

0ms

Build Failed: build daemon returned an error < failed to solve: ResourceExhausted: failed to read dockerfile: failed to create lease: write /var/lib/buildkit/runc-overlayfs/containerdmeta.db: no space left on device >

Project ID: 87ca913a-ba45-4c5c-b14b-5d17d22ae574

Service ID: 9acce407-2f44-45dd-9e5c-6ffa3bad1322 (Postgres)

Better hope my production data isn't wiped

Project ID: 44bbf8e1-e95c-4ccc-a768-91ba78438299 (taxweave)

Service ID: 842a8653-de8b-4438-821d-f52d7c430dcc (taxweave-backend)

Environment: staging (576327b0-75b0-4af4-998d-134bf1f909f7)

Latest failed deployment: 75d2f762-97fa-4e91-98a6-b53da3db8871 (commit 5ffda9d, region us-east4-eqdc4a)

Seven consecutive builds in the last hour have failed with "No space left on device" on the assigned build host. Errors progress through the pipeline as the host clears space between attempts:

4 builds (5:41–5:53 PM ET) failed at mise/Python setup: mise ERROR No space left on device (os error 28) at /tmp/railpack/mise/cache/python/pyenv

1 build (6:11 PM) failed at pip install: OSError: [Errno 28] No space left on device

2 builds (6:21 PM, 6:43 PM) failed at final image snapshot: ResourceExhausted: failed to create temp dir: mkdir /var/lib/buildkit/runc-overlayfs/snapshots/snapshots/...: no space left on device

Same error class throughout, same commit. Looks like a build host needs garbage collection or rescheduling onto a clean host.

This is a little crazy that I just switched to a paid plan with you and I haven't been able to build or deploy in over 24 hours now. I understand the higher paying customers get priority but the other people paying for a service they aren't receiving with no compensation or credit to their plans is a little crazy.

me too. i guess its just wait . wasnt sure if it was on my end or not

Hi team — Pro plan customer here, none of the F.A.Q. cases match

exactly. Dropping our info as requested.

Project ID: 05650ba0-6627-4081-aa05-d5cd4b1596d1

Service ID: b777d658-ca20-4471-bce6-738ccbb57beb (the API)

Environment ID: 7c95dafd-a744-4bf1-872c-3a5354ab6e84

Latest failed

deployment ID: 5efe84b6-4da7-4295-bf6d-eb261686f4cb

Plan: Pro

Region: us-west1

We've attempted 5 manual redeploys over the last ~2 hours

(including after the 18:06 UTC and 18:27 UTC status updates).

Every single one is scheduled on builder

"production-builderv3-us-west1-gblf" and fails in 00:02s during

"load build definition from Dockerfile". The error has actually

gotten worse over the session:

Earlier attempts (~20:35 local / 18:35 UTC):

failed to create prepare snapshot dir: failed to create temp

dir: mkdir /var/lib/buildkit/runc-overlayfs/snapshots/snapshots/

new-2506645505: no space left on device

Latest attempt (~20:49 local / 18:49 UTC):

failed to create lease: write

/var/lib/buildkit/runc-overlayfs/containerdmeta.db:

no space left on device

i.e. the builder's own metadata DB can no longer be written. The

host is fully out of disk and not recovering on its own across

redeploys.

Could you either reclaim disk on

production-builderv3-us-west1-gblf or route our service to a

healthy us-west1 builder? Our runtime service is fine (last

deployed revision still serves /health → 200), we just cannot

ship any new builds.

Blocking pre-launch hotfixes. Thank you.

Any ETA for resuming Hobby Plan deployment? My changes are on Github and auto deploy is enabled but still it is not able to deploy it.

Hi, my Postgres service is stuck in a crash loop with catatonit: failed to exec pid1: No such file or directory. The other two services (frontend and backend) are up but my database is down. As a result my application cannot be accessed now in production environments. I am on the hobby plan and I have users in production. Can you please help me bring this back up ASAP.

• Project ID: 9f7f2326-9b99-4160-afa6-7bbd4a4fc662
• Service ID: 8c3ad0a6-6c1a-4e12-a2e6-c22909830feb
• Environment ID: a1256b07-62e7-4801-a524-cf67ef88516b

I'm having the same issue.

I have the following Redis instance that it's not working:

Project: 92312c43-dd49-4cd5-8c8e-40b366fd0ad3

Service ID: 90b26469-fbf5-4b42-9cae-4fcc65ef22ad

Postgres crash loop here:

Project ID: f56c0c0a-cef1-4d7b-a6db-3be9d442afc7

Service ID: 9bc3197c-aec2-48b5-b444-7464f4278306

This is the worst customer experience ever! I just migrated all of my apps from Heroku to Railway thinking that it was a smart decision and now this... ALL of my apps were out for hours. I lost hundreds of signups. Not even a single email was sent out to customers to let us know that everything was down. And now we won't even get any sort of compensation? I am absolutely moving aways from this platform.

Although I understand that there are multiple issues you are trying to fix, I believe I speak for various people when I say that punishing the "cheaper" plans and pausing deployments when we are all trying to work on projects on different levels is unfair. I believe we deserve better than a blank we're sorry when we chose this platform compared to others.

same :/ they mark all threads as resolved and they ddnt even respond. unacceptable

I can understand the priority of higher paying customers being restored first but to not offer other PAYING tiers, hobby or not, some type of compensation for a service theyre paying for but not receiving is sad.

Hobby deploy seems to be blocked now and my only option is to upgrade?

I am still experiencing issues; the server is down, and I am unable to redeploy. Is there any update on this? Will it be resolved soon? As user @Igontijojr mentioned, we are losing a significant number of users, and our existing users are currently unable to access the server.

my db crashed, i went to restart it, its wiped. thats... great.

I have my backend, my database and Redis running in a project, and everything seems to be working fine, but Redis is failing; there’s no connection to my backend or anything like that

Yup, still working on the build queue. Working them as quickly as we get your reports.

thank you :) hope the queue isnt gonna take forever

Error: "no space left on device" on builder node before Dockerfile could be read.

Project Id: a15e5901-1471-4d5a-8d1f-655fcf5a6600

My error is this:

My production builds keep failing with ENOSPC (no space left on device) during pnpm run build. My railway.json on the main branch is set to the Railpack builder. I have a successful older deployment serving the site, but every new deployment fails on disk space. I cannot find a clear-build-cache option in the dashboard. Can you clear any stale build cache on your side and check the disk allocation on the build runner for this service?"

I am also getting same error since last night - 502 bag gateway, Can someone help me with this?

Project ID: e00a5f14-06f6-464c-a2f9-17947005057c

Service ID: 470e0b9d-558b-4c68-9632-95b5bcb33436

Issue: Service shows Online but is crashed and not responding. Cannot redeploy due to Limited Access. Bot is not running.

I have this same issue

I love Railway but honestly is one incident after the other.. Do you test the code you are writing? Do you have actual testers running tests?

I love the product but please be a bit more careful with what you ship.. You are a big company now, people are relying on you..

I was able to redeploy and it fixed the 502 Bad gateway err

No compensation for Pro plans is very disappointing, imho the Pro plan is not very Pro and the gap between Pro and Enterprise plans is way to big and the gap between Hobby and Pro way to small...

It would be much better if there was a plan between the current Pro and Enterprise with some sort of SLA and better support etc for a higher price.. a $20 plan should nopt be called "Pro" when there is not much pro about it, make a $20 Basic or starter plan and make the Pro plan $200 with better support, sla and more resources...

I am pretty sure there is a huge group of users like us that fall right in this gap, to small for enterprise to big for the current Pro plan, don't get me wrong I love Railway, but there are just to much issues and to little support options other then opening a thread.

queue takes like forever...

I'm getting a similar error when I try to deploy. If it were clear to me that upgrading to Pro would get my projects deployed then I would do it, but unfortunately the current state of Railway is very unclear. I understand that things happen and Google Cloud is also to blame but I'm baffled no email was sent out to users when these issues first started more than 24hrs ago.

Two Postgres services stuck in catatonit crash loop post-GCP outage. Multiple redeploys failed on both. Volumes need to be moved to healthy nodes. Do not reprovision data must be preserved.

esg

Project ID: deceed5e-8221-4e36-91e3-32c2328ba9d4

Service ID: b76cd947-bc1f-4a4b-a281-a289fe88ae1c

Volume ID: vol_mdennqnz59fo250f

RamPage

Project ID: 0c2cf01b-fe90-4fdc-8cfb-d0b38fcd12d6

Service ID: 60caa650-ad9f-4240-8864-6a6badb83642

Volume ID: vol_645zafxxyi1iaqee

Postgres crash loop here:

Project ID: eebb77e4-1f7a-4e82-b8be-455f3b531d96

Service ID: fd764253-08e0-4838-b589-8eb450058745

same here smh

Stuck in a Postgres crash loop also:

Project ID: 2d7dbdad-ebee-46ae-86c2-227cac22e604

Service ID: cd7c5da2-f2b0-46b2-8d9c-a70f8d676c42

We are also seeing 2 different errors that are blocking builds across some (but not all) of our services...

==================

ERROR 1:

Project ID: 16dffee8-f224-4647-ac57-f84bc0c84d40

Service ID: 7c117564-9dac-4496-b262-62d48b47f348

Service ID: d562d7c2-765a-449f-8781-0ad0cde03a61

Error: "no space left on device" during builds

==================

ERROR 2:

Project ID: 7748b63c-e160-4bed-834b-54d57b4905a7

Service ID: 45fa8a64-8cf3-43c2-9bb4-32d587372187

Error:

W: GPG error: http://deb.debian.org/debian bookworm InRelease: At least one invalid signature was encountered.

E: The repository 'http://deb.debian.org/debian bookworm InRelease' is not signed.

W: GPG error: http://deb.debian.org/debian bookworm-updates InRelease: At least one invalid signature was encountered.

E: The repository 'http://deb.debian.org/debian bookworm-updates InRelease' is not signed.

W: GPG error: http://apt.postgresql.org/pub/repos/apt bookworm-pgdg InRelease: At least one invalid signature was encountered.

E: The repository 'http://apt.postgresql.org/pub/repos/apt bookworm-pgdg InRelease' is not signed.

W: GPG error: http://deb.debian.org/debian-security bookworm-security InRelease: At least one invalid signature was encountered.

E: The repository 'http://deb.debian.org/debian-security bookworm-security InRelease' is not signed.

same issue

Matches the catatonit pid1 case from your FAQ. Tried multiple redeploys from dashboard — service exits immediately after volume mount, never reaches Postgres startup. Volume mounts successfully, data on disk should be intact.

Project ID: 5ce998e6-827c-4f7e-9d22-0ae8419f8289

Service ID: c971edd0-93f1-4bbe-babb-03d897ebecab

Environment: 5daf58ea-d37b-4bc4-a537-5defd7975ad2 (production)

Volume: vol_u38kdryvh6000fx8

Image: ghcr.io/railwayapp-templates/postgres-ssl:18

Production has been down ~15h. Upgraded Hobby → Pro today specifically to escalate this. No backups exist on this volume — please do not reprovision empty. Need volume moved to a healthy node.

Thanks.

4e3b077f-e541-4cf1-a114-1b35755b4159 this is my project id and it is queued in deployment, my authentication layer has been broken and it seems it is not working.

Project ID: 0848c2af-10a3-4248-8238-52408750a5fc

Service ID: ba90700b-b934-4304-9a4a-4476d4b8509b

Getting Error: "no space left on device" during builds

Project: (843f25ea-53e1-4304-b578-2c05d02cc530)

Error: no space left on device on builder

Node: production-builderv3-us-west1-s3s1

Multiple retries all failing with same error

My Postgres volume is full and stuck in a crash loop after the GCP outage. It completes WAL recovery but then fails with FATAL: could not write to file "pg_wal/xlogtemp.35": No space left on device on every restart attempt.

Project ID: 38c22a1f-42ce-4be3-8837-056ee222aaed

Service ID: 91ccec3f-1bc0-41bd-ad76-2ed0fb65e28d

Volume: vol_supkwekmcmp3vnyq

Please expand the volume or move it to a node with available space.

Project: 8d66af41-5f12-46ba-934a-ef7a9907840b

Service: 6fc4ac4e-ea34-4605-88bb-4a165d28745d?

environmentId: Production

Getting Error: "no space left on device" during builds Other container is able to deploy

ResourceExhausted: no space left on device

production-builderv3-us-west1-s3s1

Try now, platform reporting that all builders are cycled.

Postgres getting this error: ERROR (catatonit:2): failed to exec pid1: No such file or directory

Project: 03228f7b-5a0d-451f-9960-d83f5ca434b3

Service: ddefcd13-22d5-478f-97b6-ea69922189bb?environmentId=5ca08a48-b05f-4b6d-8442-4f9b5a96efdb

finally works.

Okay, we are fully back. For the WAL reports, we're going one by one and fixing that.

I was part of a team, but now I no longer see the team when I log in

Things are not fully back... my Postgres redeploy is still failing with the same error: ERROR (catatonit:2): failed to exec pid1: No such file or directory

Same as the person commenting immediately above: Postgres redeploy is still failing with the same error: ERROR (catatonit:2): failed to exec pid1: No such file or directory

i got my issue fixed so ignore any of my comments or request! Thank you for help

Are we going to get credited for the services that were down?

My Postgres is stuck in a crash loop with catatonit: failed to exec pid1. Already tried redeploy, still crashing.

Project ID: e65d3d2a-0d54-4520-aac4-921ae9d7452b

Service ID: 566c8d91-0606-4898-b667-930f3dd0d59e

Please move the volume to a healthy node.

My Postgres service is still offline after redeploy.

Project ID: 74e1c5a2-0489-4844-b6ca-be0c008b48df

Service ID: 3658c827-51f4-4b6d-9e3d-14371382f5f0

The service is stuck in a crash loop with:

ERROR (catatonit:2): failed to exec pid1: No such file or directory

It looks like the volume may need to be moved to a healthy node.

Please recover the Postgres service/volume. Do not delete the volume.

Thanks!

My Postgres service is still offline after redeploy.

Project ID: 3c81147d-d082-4cc9-bd54-cdc5cbb14e3f

Service ID: 4ff06ede-c82c-4016-9be3-f5d89ca8b601

The service is stuck in a crash loop with:

ERROR (catatonit:2): failed to exec pid1: No such file or directory

It looks like the volume may need to be moved to a healthy node.

Please recover the Postgres service/volume. Do not delete the volume.

Thank you

Still hitting build failures on broken builder

Project ID: 2e3c01b1-9efc-445a-bb29-d9cfe20f1209

Service ID: aa6f58db-5c5c-4811-9dbe-57b17e46579b

Environment: production (0e2dfb76-5fdd-41f2-8b4f-71778539c66a)

Builder: production-builderv3-us-west1-bbns

Issue: All builds fail on this builder — either immediately after loading Dockerfile, or partway through (non-deterministic). Build fails at various stages: sometimes at [internal] load build definition, sometimes at [builder 6/6] RUN pnpm build. No error output shown, just "Deploy failed".

Last successful deploy: May 18

Attempts: 10+ retries, always assigned to same broken builder

Disregard.