an hour ago
Service: api.vergecrm.com (backend, builds from /backend subfolder of a monorepo)
Our backend has failed to build 5 times across 2 days (Oct 7 and Oct 9), with no code changes to the backend itself between the last successful build and the first failure. Production has stayed up throughout thanks to our healthcheck gate, but we've been fully blocked from deploying backend changes for 2 days.
Two distinct failure patterns, across these deployments:
-
c2952771 (Oct 7, 14:03) - Failed. Builder: railpack-v0.40.1. Build stopped ~10s into the "prepare" step with no useful log output. Diagnosis: "Infrastructure Error," unable to pinpoint a cause, noted the same commit/config had built successfully ~16 minutes earlier.
-
fea2b7ad (Oct 7, 14:16) - Redeployed the same commit. Failed identically. Same diagnosis, same inability to pinpoint a cause.
-
62b999e6 (Oct 9, 16:27) - Redeployed again after 2 days. Failed identically. Same diagnosis.
-
4bc04af6 (Oct 9, 16:33) - After adding a RAILPACK_PYTHON_VERSION=3.13 env var to test a theory. This run switched builders entirely, to Nixpacks (matching our railway.toml's builder = "NIXPACKS" setting, which the three Railpack runs above did NOT honor). Failed differently:
pip install uv==$NIXPACKS_UV_VERSIONresolved $NIXPACKS_UV_VERSION to empty, producing an invalid requirement (uv==) and exit code 1. -
8dac45c0 (Oct 9, 16:44) - Removed the RAILPACK_PYTHON_VERSION variable to isolate whether it caused the builder switch, and redeployed. Still ran on Nixpacks (ruling out that variable as the cause of the switch), and failed with the exact same uv== / empty NIXPACKS_UV_VERSION error. This run's diagnosis also noted "the commit only changed frontend files while this service builds from the backend directory" — we're aware our service currently rebuilds on every push regardless of what changed, and are addressing that separately, but wanted to flag it since your own tooling surfaced it.
Questions:
-
Why did builder selection switch between Railpack and Nixpacks across these attempts without any explicit change on our end (other than the one variable we've now ruled out)?
-
Why is $NIXPACKS_UV_VERSION resolving to empty rather than falling back to a sensible default?
-
Is there a known issue affecting Railpack's "prepare" step around Oct 7–9 that would explain the first 3 failures?
I am happy to provide additional logs or details; my hope is to have reliable backend deploys again.
Thanks for your time.
1 Replies
an hour ago
The Railpack failure on Oct 9 at 16:27 was not an infrastructure problem: its build log shows Railpack detected Python with uv, then stopped in the prepare step with "No start command detected", which is what made prepare exit with an error. Setting the start command in the service's settings is the preferred way to give Railpack one. The Nixpacks runs did pick up a start command (uv run alembic upgrade head && uv run uvicorn ...), which suggests your railway.toml was read on those runs but not on the Railpack ones.
On builder selection, the service's saved build setting currently reads Railpack with a root directory of backend. The Railway config file does not follow the root directory, so for a monorepo it has to be referenced by its absolute path, for example /backend/railway.toml, as described in the monorepo guide. The deployment record does not show why the later Oct 9 runs resolved to Nixpacks while earlier ones used Railpack.
On the uv error, NIXPACKS_UV_VERSION is not among the variables set on this service (only three are defined), so the Nixpacks install command expanded it to nothing and pip rejected uv==.
Status changed to Awaiting User Response Railway • about 1 hour ago