Production service fails after 8 minutes
owencosans
HOBBYOP

3 hours ago

Production service stopped about 8 minutes into every deploy ("network configuration step took about 7.5 minutes"), while the same image runs fine on a sibling service

Setup: Hobby plan, Dockerfile builder, runtime V2, Python (Flask) web app. Happy to share project, service and deployment IDs privately with Railway staff.

What happens

Since Oct 31, 11:40 UTC, every deployment of one service (our production service) starts, serves traffic normally for about 8 minutes, and is then stopped by Railway: "SIGTERM received", "Stopping Container", deployment FAILED. With no earlier deployment to fall back on, the domain then returns 404 "Application not found". The dashboard's own diagnosis says the build and app startup were healthy and that "the network configuration step took about 7.5 minutes before the deployment was stopped."

The same commit and image run fine on a sibling service in the same project and environment (deployed by the same push) and on three other projects. The app does not crash; Railway stops it.

Timeline (UTC, Oct 31)

  • 02:32 deploy: ran about 9 hours, fine (the last good one).
  • 11:40 push (application code only): stopped at 11:50, about 8m41s after start.
  • 12:13 manual redeploy: stopped after about 7m54s. Railway also emailed "Build failed" at 12:21, though the build had succeeded.
  • 13:16 deploy (auto-created after we removed a stuck old deployment): about 7m58s.
  • 14:56 deploy after adding PORT=8080: started 14:57:22, SIGTERM 15:05:15 (7m53s).
  • 15:30 deploy after resetting the domains: started 15:30:34, SIGTERM 15:38:28 (7m54s).

Until the SIGTERM the logs show normal work: the app listening on 0.0.0.0:8080, background jobs completing, requests from Railway's edge (100.64.x.x) answered. The public domain answers with our sign-in page (HTTP 401, as expected) for the whole window.

What we tried (each followed by a fresh deploy; every one was stopped the same way)

  1. Redeployed from the dashboard.
  2. Removed a stuck old deployment.
  3. Added the service variable PORT=8080 (the app already listened on 8080 by default).
  4. Deleted the service's second Railway domain (explicit target port 8080) and replaced the main domain (also target port 8080) with a fresh Railway domain with no target port, renamed back to the same subdomain. It now matches the healthy sibling's single domain.

What we ruled out

  • Code or image: the same commit runs fine on the sibling service in the same environment and on three other projects.
  • Build timeout: redeploy builds finish in under a minute; the stop comes about 8 minutes after the app is up and serving.
  • Port or binding: the app listens on 0.0.0.0:8080 (Dockerfile EXPOSE 8080; PORT now also set), and public traffic is served throughout the window.
  • Health check: no healthcheck path is configured.
  • Crash or restart policy: no crash. The process runs normally until Railway's SIGTERM, and the restart policy never fires.
  • Volume: Ready, 2.5 GB of 5 GB used, mounted at /data, read fine by the app.
  • Network flows: the private network flow logs look normal (DNS, outbound 443, inbound 8080).

Possible clues (things unique to this service)

  • It is our oldest service (created July 2026) and exists in three environments: production (failing), staging (one deployment running since July 19) and demo (running since Oct 14).
  • Production's volume is also attached to this service's instance in staging. A July "railway environment new --duplicate" left the same volume ID in both environments. The healthy sibling has its own volume and lives in one environment.
  • Until today the service had two Railway domains, both with an explicit target port of 8080 (now replaced, as above).
  • An unattached Postgres volume sits in the production environment.
  • The dashboard has shown the October 21 incident banner all day ("A fix is out and we're monitoring it").

Please don't suggest deleting the volume or the staging environment. Because the volume ID is shared, deleting it from one environment once deleted it for all of them (July 2026).

Questions

  1. What is the network configuration step waiting for on this service, and why does it time out at about 7.5 minutes only here?
  2. Could the volume being referenced by two environments, or the service's age or region, block that step?
  3. Can Railway reset this service's networking state from your side? If not, what is the safe way to move this volume to a fresh service, given that staging references it too?
Solved

10 Replies

Status changed to Awaiting Railway Response Railway • about 3 hours ago


mrpenn
PRO

2 hours ago

Same here since 14:40 UTC on Oct 31 (Pro plan). One service in our staging environment fails every deploy the same way: the container runs normally, the network configuration step takes about 7.5 minutes, then the container is stopped with no reason and the deploy is marked FAILED. Five deploys of the same commit failed. The same service and image deploy fine in our production environment, and every other service in staging deploys fine. A new private network name didn’t help.


f22-pro
PRO

an hour ago

Third report, same day, same pattern. Production service in a Pro project,

no volume involved.

Three consecutive deployments of the same image, all stopped at roughly

the same age:

83aefae3 started 13:11:59Z stopped 13:20:45Z (8m46s)

7acb6640 started 13:42:39Z stopped 13:50:46Z (8m07s)

81b76111 started 13:53:33Z stopped 14:01:39Z (8m06s)

Each one built, started cleanly, passed its own startup checks and served

real traffic until the moment it stopped. The container log ends

mid-stream with only "Stopping Container" — no application error, no OOM,

and no graceful shutdown sequence from the app, so the process was killed

rather than exiting.

In the deployment events the stuck step is CONFIGURE_NETWORK: 7m37s to

8m19s on the three failures, against 2-3 seconds on healthy deployments of

this same service and of every other service in the project. After it

gives up, the deployment is marked FAILED.

Two further symptoms that may be the same root cause:

  • The previous instance is never drained. DRAIN_INSTANCES does not run, so

    the service ends up with two active deployments while configured for one

    replica, and requests land on either — ours answered 404 on the new

    routes roughly half the time.

  • railway ssh to this one service is refused ("An error occurred

    connecting to your service") while it works for every other service in

    the same project.

Setup, for completeness: no healthcheck configured, 1 replica, sleep

disabled, restart ON_FAILURE, GitHub auto-deploy off. The identical image

deploys and runs without issue in our other environment.

Workaround that restored service for us, in case it helps anyone else

here: redeploying a PREVIOUS deployment (not a new build) comes up

immediately and stays up — that path does not run CONFIGURE_NETWORK. It

does mean you are stuck on the old image until this is fixed, but it ends

the outage.


delavegaopengl
PRO

an hour ago

Same here i posted the problem 4 hours ago still no reply, we have to deploy the service every 7 minute to keep the production alive. Unbelievable


delavegaopengl

Same here i posted the problem 4 hours ago still no reply, we have to deploy the service every 7 minute to keep the production alive. Unbelievable

f22-pro
PRO

an hour ago

You don't have to redeploy every 7 minutes. Redeploying a PREVIOUS deployment — not a new build — comes up and stays up: that path doesn't run CONFIGURE_NETWORK, which is the step that hangs. You're stuck on the old image until Railway fixes this, but the outage stops.


Status changed to Awaiting User Response brody • about 1 hour ago


Status changed to Awaiting Railway Response brody • about 1 hour ago


Anonymous
HOBBY

an hour ago

The same thing happen to me today , every deploy is marked failed after 7 to 8 mins .


The same thing happen to me today , every deploy is marked failed after 7 to 8 mins .

f22-pro
PRO

an hour ago

The same previous advice helps - just keep the previous version deployed.


f22-pro

The same previous advice helps - just keep the previous version deployed.

Anonymous
HOBBY

an hour ago

yes the previous one is still up, but the new updates that I included fall off after 8 mins,


delavegaopengl
PRO

an hour ago

Our deploying resumed now


delavegaopengl

Our deploying resumed now

Anonymous
HOBBY

42 minutes ago

Confirmed, our's also resumed now,


23 minutes ago

We're looking into the deployment failures reported in this thread. Thank you for the updates, including the reports that deployments have resumed for some of you. If you're still affected, could you try one more deployment and let us know whether it completes successfully? We're continuing to investigate the failures and will follow up here when we have more information to share. I'm sorry for the disruption.


Status changed to Awaiting User Response Railway • 23 minutes ago


Status changed to Solved owencosans • 8 minutes ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...