Backend service stuck on every deploy for 3+ hours — process alive but produces zero logs, healthcheck never succeeds
bitacc
HOBBYOP

18 days ago

Hi Railway team,

I'm running into a deployment issue I can't resolve and have exhausted the debugging options available to me. I'd appreciate help from someone with backend/infrastructure access.

Project: endearing-recreation

Project ID: d03da82c-76f4-4a95-bb88-b7c6ffd52c41

Service: maritime-cyber-backend

Service ID: 79c8fcf3-d3bc-4e4c-bf8e-5fb77c53065b

Environment: production (ee4deb1d-4190-4b58-aed6-fa25797333b5)

Summary of the problem

Every deploy since approximately 14:47 UTC on 2026-08-03 has failed with the same pattern, across dozens of attempts over several hours:

  1. Docker build completes successfully.
  2. Container starts and runs our start command: python -m app.db_admin prepare && exec uvicorn app.main:app --host 0.0.0.0 --port $PORT.
  3. The first step (an Alembic database migration check) completes successfully and logs Database ready at Alembic head: .
  4. Nothing is ever logged after that point — no uvicorn startup banner, no application logs, no error, no traceback.
  5. The /health healthcheck fails with "service unavailable" on every retry attempt (tested with both 5-minute and 30-minute retry windows) until the deployment times out and is marked failed.

What I have already ruled out

  • Application code: reverted to the exact commit that deployed successfully at 14:43 UTC that same day. The identical, previously-working code still hangs the same way.
  • Database connectivity: the Alembic migration step successfully connects to and queries the same Postgres database on every single attempt, so the database itself is reachable and healthy.
  • Stale or exhausted connections: restarted the Postgres service twice and cancelled all stuck/zombie deployments on the backend service.
  • Healthcheck timeout configuration: tested with both 300s and 1800s timeouts, no difference in behavior.
  • Application logging configuration: added a raw, unbuffered sys.stderr.write() followed by an explicit flush() as the literal first line of our application's startup handler, completely bypassing Python's logging module. Still zero output.
  • Process topology: changed our container's CMD to exec uvicorn so it replaces the shell and runs as PID 1 directly, instead of running as a child process under sh -c "... && ...", in case there was a log-capture gap specific to non-PID-1 processes. No change in behavior.

Additional evidence

Using your in-dashboard AI agent (before running out of usage credits for it), I was able to get live process-level inspection of one of the stuck containers (deployment 1f11c409-3107-476e-8ff7-ab364855b745). It found:

  • The application process was alive and running (not crashed): PID 9, uvicorn, 5 threads, approximately 221MB resident memory, normal open file descriptors.
  • The process was blocked in epoll_wait, which is the normal idle state for a running async event loop.

So the process reaches a stable, running state — it simply never produces any output and never responds to the healthcheck.

I also noticed build times increased noticeably over the course of these attempts (roughly 1 minute for early attempts that day, up to 4 minutes for later ones), and build logs reference the builder node builder-oersqr, in case that's relevant.

What I'd like help checking

  1. What value of the PORT environment variable was actually injected into one of these containers, and does it match the port the process bound to and the port the healthcheck is probing?
  2. Whether there's a known issue with log delivery/capture for this project, region, or builder node around this time window.
  3. Whether builder builder-oersqr or the underlying host it runs on is in a degraded state.

Recent affected deployment IDs for reference:

  • e40a7a3e-6e38-44a8-9c9d-48d1e202828f
  • 1f11c409-3107-476e-8ff7-ab364855b745
  • 21e65b08-60f7-431a-9b4d-593cf611f360

I'm happy to trigger a fresh deployment on request if it would help catch the issue live, and can provide repository access if that's useful.

Thank you for your help — this has taken down our production backend for several hours and I'd appreciate a prompt response.

Best regards,

$10 Bounty

5 Replies

Railway
BOT

18 days ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway 18 days ago


sardarmudassaralikhan
FREE

18 days ago

To resolve this deadlock and pass the healthcheck successfully, implement the following end-to-end working solution:

  1. Decouple Database Migrations via Railway's Release Phase

Running blocking pre-flight scripts (python -m app.db_admin prepare) directly inside your main container start command can cause race conditions with the platform's initial port-binding and healthcheck window.

Move the database migration command out of the start command entirely and into a dedicated Release Phase configuration (or run it natively via Railway's project settings as a pre-deploy build step). This guarantees that when your web container boots, it runs only the ASGI server as PID 1 immediately.

  1. Fix Container Entrypoint & Port Binding Configuration

Update your service's start command (or Procfile/Dockerfile entrypoint) to ensure standard shell-form expansion of the dynamic $PORT variable. Replace your current start command with a robust shell script pattern that guarantees correct execution and stream flushing:

Bash

python -u -m app.db_admin prepare && exec uvicorn app.main:app --host 0.0.0.0 --port ${PORT:-8000}

python -u: Forces unbuffered binary stdout and stderr streams so that Python logs flush immediately to the host logging pipeline without getting trapped in buffer blocks.

${PORT:-8000}: Provides a safe fallback port if the platform variable injection ever experiences a temporary evaluation race.

  1. Add an Explicit Startup Event Hook for Fast Probing

To ensure Uvicorn answers the /health probe instantly upon binding to the network socket, implement an explicit application startup event in your FastAPI app.main file so the health router responds immediately even before background worker threads complete warm-up:

Python

from fastapi import FastAPI

app = FastAPI()

@app.on_event("startup")

async def startup_event():

# Ensures the app signals readiness immediately upon socket open

pass

@app.get("/health")

async def health_check():

return {"status": "healthy"}

4. Force a Clean Cache Redeploy

Once the updated configuration is saved:

Trigger a fresh deployment by pushing an empty commit or utilizing the Redeploy option in the Railway dashboard.

This forces the platform to provision a clean container instance away from the degraded builder node (builder-oersqr), clearing out any hung socket states and re-establishing correct log streaming pipes.


18 days ago

I have seen similar issues where someone trying to run two commands but only the first one executed

Try wrap your start command in a shell like this

sh -c "python -m app.db_admin prepare && exec uvicorn app.main:app --host 0.0.0.0 --port $PORT"


kasayka1999
PRO

17 days ago

  1. try: uvicorn app.main:app --host 0.0.0.0 --port 8000

(sometimes i have also problem with $PORT.)

  1. Confirm Uvicorn is listening on 0.0.0.0:8000.

  2. Make sure /health is accessible without authentication.


brezzy1337
HOBBYTop 10% Contributor

8 days ago

Hey! It's been a while since you posted, so hopefully you've found this already. I was able to reproduce your failure on a test service, and I think a good place to start is going over the questions you had.

Service checks

  1. In my test $PORT is injected as PORT=8080, uvicorn bound 0.0.0.0:8080, and the probe reached it from 100.64.0.2 with 200 OK. --host 0.0.0.0 is right, and you do not need --host :: for the healthcheck. Worth saying since it came up above: don't hardcode 8000 or use ${PORT:-8000} — that would create the very mismatch you were asking about.

  2. I ran one image twice, healthy and never-healthy, printing to stdout and stderr every 2 seconds for three minutes. Both delivered all 90 lines, in both panes, during the failure and after it was marked failed. I wouldn't lean hard on the first second of output either way, but a three-minute silence is the process not writing rather than Railway not delivering.

  3. builder-oersqr is not in this path. Your builds succeeded every time, and the failure happens after the container starts. sardarmudassaralikhan is right that nothing binds a port until the pre-flight step finishes — but your Alembic step completes and prints, so moving it out will not change this.

So to frame the issue: something in your startup path blocks and never returns, so uvicorn never reaches the point where it prints its banner or opens a socket. The healthcheck then gets connection refused, which Railway reports as service unavailable.

Two things in your own post support that. Uvicorn running on … prints last, because the socket is opened last — so no banner means nothing is listening, which is why 300s and 1800s behaved identically. And the same commit worked at 14:43 and hung at 14:47, so whatever changed is outside the image. You restarted Postgres twice in that window, which fits: a startup call that dials the database returned at 14:43 and stopped returning at 14:47.

Your Alembic step working doesn't clear the database either, incidentally. Alembic runs on a sync driver (psycopg2) and your app on an async engine (asyncpg), often against different URLs — two separate connections, so one can succeed every time while the other hangs.


brezzy1337

Hey! It's been a while since you posted, so hopefully you've found this already. I was able to reproduce your failure on a test service, and I think a good place to start is going over the questions you had. #### Service checks 1. In my test `$PORT` is injected as `PORT=8080`, uvicorn bound `0.0.0.0:8080`, and the probe reached it from `100.64.0.2` with `200 OK`. `--host 0.0.0.0` is right, and you do not need `--host ::` for the healthcheck. Worth saying since it came up above: don't hardcode `8000` or use `${PORT:-8000}` — that would create the very mismatch you were asking about. 2. I ran one image twice, healthy and never-healthy, printing to stdout and stderr every 2 seconds for three minutes. Both delivered all 90 lines, in both panes, during the failure and after it was marked failed. I wouldn't lean hard on the first second of output either way, but a three-minute silence is the process not writing rather than Railway not delivering. 3. **`builder-oersqr` is not in this path.** Your builds succeeded every time, and the failure happens after the container starts. sardarmudassaralikhan is right that nothing binds a port until the pre-flight step finishes — but your Alembic step completes and prints, so moving it out will not change this. So to frame the issue: something in your startup path blocks and never returns, so uvicorn never reaches the point where it prints its banner or opens a socket. The healthcheck then gets connection refused, which Railway reports as `service unavailable`. Two things in your own post support that. `Uvicorn running on …` prints **last**, because the socket is opened last — so no banner means nothing is listening, which is why 300s and 1800s behaved identically. And the same commit worked at 14:43 and hung at 14:47, so whatever changed is outside the image. You restarted Postgres twice in that window, which fits: a startup call that dials the database returned at 14:43 and stopped returning at 14:47. Your Alembic step working doesn't clear the database either, incidentally. Alembic runs on a **sync** driver (`psycopg2`) and your app on an **async** engine (`asyncpg`), often against different URLs — two separate connections, so one can succeed every time while the other hangs.

brezzy1337
HOBBYTop 10% Contributor

8 days ago

The best hint we have is where your logs stop, which you've narrowed to just after the Alembic migration check. So let's make the process tell you where it is stuck.

Fix

  1. Add these two lines at the very top of app/main.py, above every other import:

    
    import faulthandler
    
    faulthandler.dump_traceback_later(15, repeat=True, exit=False)
    
  2. Deploy. Every 15 seconds the deploy log gets a traceback of where it is stuck. Mine, from a container blocking on a connect at module level:

    
    Timeout (0:00:15)!
    
    Thread 0x00007fe2da736740 (most recent call first):
    
      File "/root/.nix-profile/lib/python3.12/socket.py", line 850 in create_connection
    
      File "/app/hangmod.py", line 5 in <module>
    
      File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
    
      File "<frozen importlib._bootstrap_external>", line 995 in exec_module
    
      File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
    

    Read down past the standard-library frames: the first File line pointing at your own code is the one to fix/app/hangmod.py above, not socket.py. With asyncio the top frame is usually the event loop, so it is never the interesting one. The <frozen importlib._bootstrap> frames underneath mean it is blocking at import, which also tells you your startup handler never ran — and explains why your raw stderr write printed nothing. If those frames are absent and you see your own startup function instead, the block is inside the handler.

    I tested this on a deployment that never becomes healthy — it still reports.

  3. Fix according to what that line names:

    • Your own module-level code — an engine created at import, a pool warm-up, a client, a secrets fetch. Move it into the startup handler and bound it:

      
      @asynccontextmanager
      
      async def lifespan(app: FastAPI):
      
          app.state.engine = create_async_engine(DATABASE_URL, connect_args={"timeout": 10})
      
          await asyncio.wait_for(app.state.engine.connect(), timeout=15)
      
          yield
      
      app = FastAPI(lifespan=lifespan)
      
    • A third-party module that connects while being imported. You cannot move code you do not own, so stop importing it at module scope — move import x inside the function that uses it — or pass that library an explicit connect timeout.

    • Already inside your startup handler. Leave it there and wrap the call: await asyncio.wait_for(..., timeout=15).

  4. If that line is a database connect, compare the URL your app builds with the one app/db_admin.py hands Alembic. You are looking for a different host, not a different driver prefix — postgresql+asyncpg:// vs postgresql:// against the same host is normal. A public or proxy host on one side and *.railway.internal on the other, or a hardcoded address, is the discrepancy worth finding.

Add the timeout even once the endpoint is healthy again. An unbounded network call at import turns any slow dependency into this exact failure: live process, no logs, healthcheck that can never pass. With one, you get a traceback and a deploy that fails in seconds instead of 30 minutes.


Welcome!

Sign in to your Railway account to join the conversation.

Loading...