Production Postgres service crash-looping — building wrong source (app code, not postgres image)
taxman914
PROOP

a month ago

Production database is down. My Postgres service is crash-looping because it is

building and running my APP'S code instead of the Postgres image, even though

its Source setting shows the correct image.

PROJECT DETAILS

  • Project: CPA-Hub (production environment)

  • Project ID: f5160e48-5a4c-4fed-ac13-669ff14af51b

  • Environment ID: b483c890-355f-4a95-aaad-a04a1fd51335

  • Postgres service ID: 74856220-35d1-4200-9a1c-0452182d333d

  • Data volume: postgres-volume (vol_y0y1qvkq2e4vh4jt), attached at

    /var/lib/postgresql/data, ~0.5GB used

THE PROBLEM

The Postgres service's Settings > Source shows the correct image

(ghcr.io/railwayapp-templates/postgres-ssl:17). However, every deployment —

including brand-new ones I've triggered — actually builds and runs

"node src/index.js" from my app repo (taxman914/cpahub) and crash-loops on a

missing app environment variable (APP_SECRET). The service config and the

actual build source are inconsistent.

Deployment logs repeatedly show:

pbi-hub@1.0.0 start

node src/index.js

Environment Validation Failed: Missing required: APP_SECRET

A new deployment created ~14:44 UTC did the same thing — same data volume

re-attached (new bind-mount host path, same volume ID vol_y0y1qvkq2e4vh4jt),

but still ran node and crash-looped. So redeploying does not fix it; new

deployments keep building the wrong source.

TIMELINE / SUSPECTED CAUSE

This began during today's GitHub Actions / Railway API incident (partial

failures reported ~14:14 UTC). I suspect a deployment event during that

instability caused the Postgres service to pick up my app's build config.

CURRENT STATE

  • My app service (cpahub) is UP but has no working database — all queries hang.

  • The Postgres data volume appears INTACT and re-attaches on every deployment,

    but is never opened by a real Postgres process (the node container dies at

    env validation before touching it).

WHAT I NEED

Please restore the Postgres service so it actually deploys the

postgres-ssl:17 image against my existing postgres-volume

(vol_y0y1qvkq2e4vh4jt), without risking the data on that volume.

I have NOT modified, detached, or reset any volume, and I will not touch the

volume while this is open. Please advise before any action that could affect

the data.

Solved

9 Replies

Status changed to Awaiting Railway Response Railway about 1 month ago


Try clicking on your database, press Ctrl/Cmd + K, and redeploy source image.

If that doesn't work, you'll need to create a new Postgres service, delete the old service (not the volume), and drag the old volume over the new Postgres service to swap volumes.


taxman914
PROOP

a month ago

AWESOME....we are back. Thank you for the amazingly fast response and will note that for future reference.


Status changed to Solved Railway about 1 month ago


taxman914
PROOP

a month ago

Is this possible? 1. Ask Railway to clear the stale source.repo permanently. This is the one I'd actually prioritize, because it's a latent landmine. "Redeploy source image" worked around the conflict — it forced the image this once. But if the stale source.repo/source.branch fields are still set on the Postgres service, a future ordinary redeploy (or an auto-deploy triggered by something) could rebuild your app onto Postgres again and drop you right back into this outage. So reply on the thread: confirm the source-image redeploy fixed it, thank them, and explicitly ask them to clear the stale source.repo/source.branch fields so only source.image remains — otherwise this recurs. That closes the actual root cause, not just the symptom.


Status changed to Awaiting Railway Response Railway about 1 month ago


taxman914
PROOP

a month ago

Update: "Redeploy source image" (via Cmd+K) worked — the Postgres service

booted the postgres-ssl:17 image against the existing volume, and all my data

is intact. Thank you.

One concern before I consider this closed: I believe the underlying cause —

the stale source.repo / source.branch fields conflicting with source.image —

is still present on the service. "Redeploy source image" worked around it by

forcing the image this time, but if those repo/branch fields are still set,

I'm worried a future ordinary redeploy (or an auto-deploy) could rebuild my app

onto the Postgres service again and repeat this outage.

Could you please clear the stale source.repo / source.branch fields on the

Postgres service (service ID 74856220-35d1-4200-9a1c-0452182d333d) so that only

source.image remains? I'd like to make sure this can't recur on the next

redeploy.

Thanks again for the quick help.


Status changed to Awaiting Railway Response Railway about 1 month ago


taxman914

Update: "Redeploy source image" (via Cmd+K) worked — the Postgres service booted the postgres-ssl:17 image against the existing volume, and all my data is intact. Thank you. One concern before I consider this closed: I believe the underlying cause — the stale source.repo / source.branch fields conflicting with source.image — is still present on the service. "Redeploy source image" worked *around* it by forcing the image this time, but if those repo/branch fields are still set, I'm worried a future ordinary redeploy (or an auto-deploy) could rebuild my app onto the Postgres service again and repeat this outage. Could you please clear the stale source.repo / source.branch fields on the Postgres service (service ID 74856220-35d1-4200-9a1c-0452182d333d) so that only source.image remains? I'd like to make sure this can't recur on the next redeploy. Thanks again for the quick help.

It's possible that you ran railway up via CLI into the Postgres service.

If this happens again, then I'd recommend creating a new Postgres service and migrating the old volume over.


taxman914
PROOP

a month ago

Sorry, I have Claude working with me right now and it is 'contingency planning' My database is back up and running just fine


Status changed to Awaiting Railway Response Railway about 1 month ago


sam-a
EMPLOYEE

a month ago

Glad you're back up! The stale repo binding is worth clearing so this can't recur. On the Postgres service, go to Settings > Service Source, disconnect the current source, then reconnect it by selecting the Docker image (postgres-ssl:17). That removes the repo/branch fields permanently.


Status changed to Awaiting User Response Railway about 1 month ago


Status changed to Solved sam-a about 1 month ago


sam-a

Glad you're back up! The stale repo binding is worth clearing so this can't recur. On the Postgres service, go to Settings > Service Source, disconnect the current source, then reconnect it by selecting the Docker image (postgres-ssl:17). That removes the repo/branch fields permanently.

taxman914
PROOP

a month ago

Thanks, but "redeploy" has now failed three times with the same generic container-creation error and no build or deploy logs, so I'd like to get past the retry loop and have someone look at the actual container spec.

Context: this is a volume-backed Postgres service (postgres-ssl:17). Per your earlier guidance, I disconnected the service source and reconnected it as the Docker image to clear the stale source.repo/branch binding. The failures started immediately after that reconnect. The prior deployment — same image, same config — is still running fine, so the image itself is not the problem.

Because it fails at container creation before any logs, this looks like an infrastructure/spec issue, not my configuration. Can you please:

Confirm the volume is able to detach from the still-running prior deployment so the new container can mount it. If this is a single-attach volume conflict, that would explain a no-logs container-creation failure.

Diff the container spec generated by the source reconnect against my last successful deployment — did the disconnect/reconnect drop or alter the volume mount path (/var/lib/postgresql/data), any Postgres env vars, or a start command?

Confirm there's no platform/scheduler or region capacity issue causing the container creation to fail.

Tell me the safe procedure to apply the corrected image-only source without risking the currently-running deployment — I do not want to lose the deployment that's keeping my database online.

Please do not take any destructive action on the volume — production data lives there. Failed deploy IDs: [FAILED_DEPLOYMENT_ID]; last known-good deployment still serving: [GOOD_DEPLOYMENT_ID].


Status changed to Awaiting Railway Response Railway about 1 month ago


Railway
BOT

a month ago

The failed deployments are trying to pull docker.io/library/postgres-ssl:17 (Docker Hub), which doesn't exist. The correct image is ghcr.io/railwayapp-templates/postgres-ssl:17 (GitHub Container Registry). When you reconnected the source, the image reference was likely saved without the full registry prefix. Your volume and its ~485 MB of data are intact. To fix this, disconnect the source on the Postgres service again and reconnect it using the full image path ghcr.io/railwayapp-templates/postgres-ssl:17. Your currently running deployment will keep serving until a new one succeeds and replaces it.


Status changed to Awaiting User Response Railway about 1 month ago


Railway
BOT

a month ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway about 1 month ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...