a month ago
Project: [a1706405-33ee-4a83-bf17-a55089780995] · Service: [dcea357a-a193-4e20-be28-50a1ac513b8f] (halo-foundry)
preDeployCommand is set via railway.toml (node dist/migrate.js) and confirmed live in Service Settings.
On deployment 473c438c (Jul 17, 23:39 PDT, GitHub-triggered from merged main), the pre-deploy step never appeared in the deployment's step list, an anonymous container ran 23:39:54–23:39:56 with zero log output, and the deployment succeeded. The command is a migration runner that logs unconditionally on every execution path (including zero-pending), so it never executed.
The immediately prior deployment eeb30770 (23:27, byte-identical railway.toml) correctly rendered Deploy › Pre-deploy command ✗ when the command failed, and pre-deploy executed correctly on four other deploys the same evening on this service.
The only distinguishing feature of 473c438c: its image digest (sha256:884e734bf4f533f5e64e13e57ab90c9d8602ba99ba2dfabe6a6f40f202b5621f) was byte-identical to the then-serving deployment ccf1e237 — the source tree was identical after a revert, so the build fully cache-hit. Skipped Builds feature flag is OFF, Watch Paths empty.
Note: two subsequent variable-change-triggered deploys with image digests identical to the serving deployment did execute pre-deploy normally — the skip we observed was on a push-triggered deploy specifically.
Questions: (1) Can pre-deploy scheduling be skipped when a push-triggered deployment's image digest matches the active deployment? (2) If not, what would cause the pre-deploy step to be absent from a deployment's plan with valid config? (3) Separately: can you confirm that on healthcheck failure/timeout the previous deployment always keeps serving? We've observed this once empirically but the docs only imply it via the zero-downtime mechanic.
1 Replies
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • about 1 month ago
16 days ago
A build cache hit or an image digest matching the active deployment is not a documented reason to skip a configured pre-deploy command. Railway documents pre-deploy as a separate container step after the build and before the application deploy. It also states that a non-zero pre-deploy exit prevents the deployment from proceeding.
The decisive check is not the service's current Settings page. Open deployment 473c438c itself and inspect its Deployment Details. Railway shows the effective settings that each individual deployment went out with, including a file icon when a value came from railway.toml.
That gives a clean split:
- If
node dist/migrate.jsis absent from that deployment's effective configuration, the failure happened while resolving the deployment snapshot/config. Check the configured Config File Path and therailway.tomlcontained in that deployment's exact source snapshot. - If the command is present in the effective configuration but there is no pre-deploy step and the migration-side marker is absent, the deployment contradicted the documented lifecycle. That is a Railway scheduler/control-plane incident tied to that deployment ID, not an image-cache behavior an application can fix.
- If the command is present and the two-second anonymous container is the pre-deploy container, but its unconditional output is missing, then distinguish command execution from log ingestion using the migration table or another idempotent database marker. Container logs alone are not enough to prove which of those two failed.
For future correlation, put a tiny checked-in wrapper around the migration command:
#!/bin/sh
set -eu
printf 'predeploy start deployment=%s commit=%s\n' \
"${RAILWAY_DEPLOYMENT_ID:-unset}" \
"${RAILWAY_GIT_COMMIT_SHA:-unset}"
node dist/migrate.js
printf 'predeploy complete deployment=%s\n' \
"${RAILWAY_DEPLOYMENT_ID:-unset}"Then set preDeployCommand to the wrapper. Railway provides both IDs to deployments, so the log and migration record can be matched to one deployment without printing any secrets. Keep the migration idempotent and fail non-zero on an actual migration error; do not duplicate it in the Start Command, where overlapping replicas could run it concurrently.
For the healthcheck question: when a healthcheck is configured, Railway says the new deployment becomes active only after it returns HTTP 200; only then is the previous deployment made inactive. If the healthcheck times out, the new deployment is marked failed, so the previous active deployment should continue serving. The documented exception is a service with an attached volume: Railway cannot keep two deployments mounted to that volume simultaneously, so zero-downtime overlap is not available there.
Official references:
- Pre-deploy lifecycle and failure behavior: https://docs.railway.com/deployments/pre-deploy-command
- Per-deployment effective Config-as-Code view: https://docs.railway.com/config-as-code
- Railway-provided deployment and Git commit variables: https://docs.railway.com/variables/reference
- Healthcheck activation and volume caveat: https://docs.railway.com/deployments/healthchecks
Given the evidence in the question, Railway still needs to inspect the control-plane trace for 473c438c if the effective deployment details contain the command. The matching image digest alone does not explain the omission under the documented behavior.