a month ago
Production database is down. My Postgres service is crash-looping because it is
building and running my APP'S code instead of the Postgres image, even though
its Source setting shows the correct image.
PROJECT DETAILS
-
Project: CPA-Hub (production environment)
-
Project ID: f5160e48-5a4c-4fed-ac13-669ff14af51b
-
Environment ID: b483c890-355f-4a95-aaad-a04a1fd51335
-
Postgres service ID: 74856220-35d1-4200-9a1c-0452182d333d
-
Data volume: postgres-volume (vol_y0y1qvkq2e4vh4jt), attached at
/var/lib/postgresql/data, ~0.5GB used
THE PROBLEM
The Postgres service's Settings > Source shows the correct image
(ghcr.io/railwayapp-templates/postgres-ssl:17). However, every deployment —
including brand-new ones I've triggered — actually builds and runs
"node src/index.js" from my app repo (taxman914/cpahub) and crash-loops on a
missing app environment variable (APP_SECRET). The service config and the
actual build source are inconsistent.
Deployment logs repeatedly show:
pbi-hub@1.0.0 start
node src/index.js
Environment Validation Failed: Missing required: APP_SECRET
A new deployment created ~14:44 UTC did the same thing — same data volume
re-attached (new bind-mount host path, same volume ID vol_y0y1qvkq2e4vh4jt),
but still ran node and crash-looped. So redeploying does not fix it; new
deployments keep building the wrong source.
TIMELINE / SUSPECTED CAUSE
This began during today's GitHub Actions / Railway API incident (partial
failures reported ~14:14 UTC). I suspect a deployment event during that
instability caused the Postgres service to pick up my app's build config.
CURRENT STATE
-
My app service (cpahub) is UP but has no working database — all queries hang.
-
The Postgres data volume appears INTACT and re-attaches on every deployment,
but is never opened by a real Postgres process (the node container dies at
env validation before touching it).
WHAT I NEED
Please restore the Postgres service so it actually deploys the
postgres-ssl:17 image against my existing postgres-volume
(vol_y0y1qvkq2e4vh4jt), without risking the data on that volume.
I have NOT modified, detached, or reset any volume, and I will not touch the
volume while this is open. Please advise before any action that could affect
the data.
9 Replies
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
Try clicking on your database, press Ctrl/Cmd + K, and redeploy source image.
If that doesn't work, you'll need to create a new Postgres service, delete the old service (not the volume), and drag the old volume over the new Postgres service to swap volumes.
a month ago
AWESOME....we are back. Thank you for the amazingly fast response and will note that for future reference.
Status changed to Solved Railway • about 1 month ago
a month ago
Is this possible? 1. Ask Railway to clear the stale source.repo permanently. This is the one I'd actually prioritize, because it's a latent landmine. "Redeploy source image" worked around the conflict — it forced the image this once. But if the stale source.repo/source.branch fields are still set on the Postgres service, a future ordinary redeploy (or an auto-deploy triggered by something) could rebuild your app onto Postgres again and drop you right back into this outage. So reply on the thread: confirm the source-image redeploy fixed it, thank them, and explicitly ask them to clear the stale source.repo/source.branch fields so only source.image remains — otherwise this recurs. That closes the actual root cause, not just the symptom.
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
Update: "Redeploy source image" (via Cmd+K) worked — the Postgres service
booted the postgres-ssl:17 image against the existing volume, and all my data
is intact. Thank you.
One concern before I consider this closed: I believe the underlying cause —
the stale source.repo / source.branch fields conflicting with source.image —
is still present on the service. "Redeploy source image" worked around it by
forcing the image this time, but if those repo/branch fields are still set,
I'm worried a future ordinary redeploy (or an auto-deploy) could rebuild my app
onto the Postgres service again and repeat this outage.
Could you please clear the stale source.repo / source.branch fields on the
Postgres service (service ID 74856220-35d1-4200-9a1c-0452182d333d) so that only
source.image remains? I'd like to make sure this can't recur on the next
redeploy.
Thanks again for the quick help.
Status changed to Awaiting Railway Response Railway • about 1 month ago
taxman914
Update: "Redeploy source image" (via Cmd+K) worked — the Postgres service booted the postgres-ssl:17 image against the existing volume, and all my data is intact. Thank you. One concern before I consider this closed: I believe the underlying cause — the stale source.repo / source.branch fields conflicting with source.image — is still present on the service. "Redeploy source image" worked *around* it by forcing the image this time, but if those repo/branch fields are still set, I'm worried a future ordinary redeploy (or an auto-deploy) could rebuild my app onto the Postgres service again and repeat this outage. Could you please clear the stale source.repo / source.branch fields on the Postgres service (service ID 74856220-35d1-4200-9a1c-0452182d333d) so that only source.image remains? I'd like to make sure this can't recur on the next redeploy. Thanks again for the quick help.
a month ago
It's possible that you ran railway up via CLI into the Postgres service.
If this happens again, then I'd recommend creating a new Postgres service and migrating the old volume over.
a month ago
Sorry, I have Claude working with me right now and it is 'contingency planning' My database is back up and running just fine
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
Glad you're back up! The stale repo binding is worth clearing so this can't recur. On the Postgres service, go to Settings > Service Source, disconnect the current source, then reconnect it by selecting the Docker image (postgres-ssl:17). That removes the repo/branch fields permanently.
Status changed to Awaiting User Response Railway • about 1 month ago
Status changed to Solved sam-a • about 1 month ago
sam-a
Glad you're back up! The stale repo binding is worth clearing so this can't recur. On the Postgres service, go to Settings > Service Source, disconnect the current source, then reconnect it by selecting the Docker image (postgres-ssl:17). That removes the repo/branch fields permanently.
a month ago
Thanks, but "redeploy" has now failed three times with the same generic container-creation error and no build or deploy logs, so I'd like to get past the retry loop and have someone look at the actual container spec.
Context: this is a volume-backed Postgres service (postgres-ssl:17). Per your earlier guidance, I disconnected the service source and reconnected it as the Docker image to clear the stale source.repo/branch binding. The failures started immediately after that reconnect. The prior deployment — same image, same config — is still running fine, so the image itself is not the problem.
Because it fails at container creation before any logs, this looks like an infrastructure/spec issue, not my configuration. Can you please:
Confirm the volume is able to detach from the still-running prior deployment so the new container can mount it. If this is a single-attach volume conflict, that would explain a no-logs container-creation failure.
Diff the container spec generated by the source reconnect against my last successful deployment — did the disconnect/reconnect drop or alter the volume mount path (/var/lib/postgresql/data), any Postgres env vars, or a start command?
Confirm there's no platform/scheduler or region capacity issue causing the container creation to fail.
Tell me the safe procedure to apply the corrected image-only source without risking the currently-running deployment — I do not want to lose the deployment that's keeping my database online.
Please do not take any destructive action on the volume — production data lives there. Failed deploy IDs: [FAILED_DEPLOYMENT_ID]; last known-good deployment still serving: [GOOD_DEPLOYMENT_ID].
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
The failed deployments are trying to pull docker.io/library/postgres-ssl:17 (Docker Hub), which doesn't exist. The correct image is ghcr.io/railwayapp-templates/postgres-ssl:17 (GitHub Container Registry). When you reconnected the source, the image reference was likely saved without the full registry prefix. Your volume and its ~485 MB of data are intact. To fix this, disconnect the source on the Postgres service again and reconnect it using the full image path ghcr.io/railwayapp-templates/postgres-ssl:17. Your currently running deployment will keep serving until a new one succeeds and replaces it.
Status changed to Awaiting User Response Railway • about 1 month ago
a month ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • about 1 month ago