2 months ago
We are experiencing 16+ consecutive Railway deployment failures for our production service. The currently running production deployment remains healthy, but every new deployment is failing during BUILD_IMAGE before any Railpack build steps execute.
Service: Sportbook-me-
Service ID: eff8518d-9b0f-473b-8e9d-ac01713a37e8
Failed Deployment: 0f60c94f-a353-4978-ae98-7c4e3e174183
Healthy Production Deployment: 45a8855c-7453-42ec-96b1-e5eadda33dac
Known-good commit: 6fd72f4c565c66c2a580657b9a7a37e49f152074
Builder: builder-rysztn
Railpack: 0.38.0
Python: 3.13.15
Failure window: approximately 2026-08-26 21:51 UTC onward
SNAPSHOT_CODE completes successfully and Railpack detects the Python provider. BUILD_IMAGE then fails almost immediately. The complete build log contains only:
scheduling build on Metal builder "builder-rysztn"
scheduling build on Metal builder "builder-rysztn"
There is no Railpack build output, pip output, application error, dependency error, or traceback. The BUILD_IMAGE stage terminates before the normal build steps begin.
The same source/configuration successfully built earlier on the same builder. The active production deployment is still online and /health returns HTTP 200.
We have reproduced the same silent BUILD_IMAGE failure across 16+ consecutive deployment attempts.
Please investigate Railway's builder infrastructure for this service, particularly builder-rysztn. Please check internal builder logs/metrics for crashes, OOM termination, driver initialization failure, or lost logging/artifact connectivity. If appropriate, please drain/restart the affected builder or rebalance this service to a healthy builder.
Please also advise whether it is safe for us to retry deployment once the infrastructure issue has been addressed. We are intentionally not making further production changes while this remains unresolved.
Additional context: Railway Central Station automatically identified multiple similar recent reports when this ticket title was entered, including “Repeated pre-Docker BUILD_IMAGE failure on the same Railway Metal builder” posted approximately 16 hours ago. This may indicate the failure is not isolated to this service.
5 Replies
2 months ago
Having looked into this, the issue appears to be in your application code or configuration rather than the Railway platform itself, which puts it outside what Railway support can resolve directly.
This is exactly the kind of problem the Railway community is good at, so we'd like to open your thread as a community bounty. Railway pays a bounty to the community member who solves it, and threads like this usually get picked up quickly.
Opening it makes this entire thread public, including everything already posted. Nothing becomes public until you decide. Use the buttons below.
- Open to the community - Before you click, take a moment to edit or remove anything you'd rather not share. The thread becomes publicly visible right away.
- Keep it private and close the thread - Nothing becomes public and the thread closes.
Status changed to Awaiting User Response Railway • about 2 months ago
2 months ago
Thank you, but I still need assistance from a Railway support engineer.
The conclusion that this is application code/configuration does not appear consistent with the deployment evidence.
The affected deployment completes SNAPSHOT_CODE, Railpack detects Python, and then BUILD_IMAGE terminates within seconds with only:
scheduling build on Metal builder "builder-rysztn"
repeated twice.
There is no Python traceback, dependency-resolution error, configuration error, pip error, application startup error, or other diagnostic output.
More importantly, the source is bit-identical to commit 6fd72f4, which successfully built earlier using the same Railpack version, Python version, service configuration, and builder.
We also reproduced the same silent pre-build failure across 16+ deployment attempts.
Before categorizing this as an application issue, could a Railway engineer please review the internal BUILD_IMAGE / Metal-builder records for deployment:
0f60c94f-a353-4978-ae98-7c4e3e174183
Service:
eff8518d-9b0f-473b-8e9d-ac01713a37e8
Specifically, please identify what application or configuration error Railway believes caused BUILD_IMAGE to terminate. If there is an application-level error, please provide the actual diagnostic/error record so we can correct it.
We do not want to make the thread public until Railway engineering has reviewed this evidence.
Status changed to Awaiting Railway Response Railway • about 2 months ago
2 months ago
This still looks like an application-level problem, so Railway support can't take it further, but the community can. The buttons below are still live: open the thread up as a public bounty after editing out anything sensitive, or keep it private and close it.
Status changed to Awaiting User Response Railway • about 2 months ago
2 months ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • about 2 months ago
2 months ago
Additional technical context — what we need help solving
We are looking for help identifying why new deployments of this service are failing during Railway's BUILD_IMAGE stage even though the currently deployed version is healthy.
Current production status
Existing production deployment is ONLINE and healthy.
/health returns HTTP 200.
We do not want to disturb that working deployment while troubleshooting.
Failure behavior
We have experienced 16+ consecutive failed deployment attempts.
SNAPSHOT_CODE completes successfully.
Railpack detects the application as Python.
Python 3.13.15 is resolved.
Railpack version is 0.38.0.
BUILD_IMAGE then fails within only a few seconds.
There is no pip install output, Python traceback, dependency-resolution error, syntax error, or application startup error.
The only build-log output is:
scheduling build on Metal builder "builder-rysztn"
followed by the same scheduling message again.
Important comparison
The same application source/configuration successfully built earlier. The currently running production deployment was built successfully from known-good commit 6fd72f4.
We have also tested deployment attempts using source matching that known-good state, but the new builds still terminate during BUILD_IMAGE.
What we need from the community
We are specifically trying to determine:
What can cause Railway BUILD_IMAGE to fail before normal Railpack build output appears?
Is there a Railpack/Railway configuration or repository artifact that could cause this silent failure?
What diagnostic should we run next to distinguish an application/repository problem from a builder problem?
Is there a safe way to force a clean Railpack build/rebuild without jeopardizing the currently healthy production deployment?
Please do not suggest destructive production changes without explaining the impact first. This is a live production service, and the existing deployment is healthy.
Our immediate objective is simply to restore the ability to successfully build and deploy the existing backend again. Once that is working, we have a small production configuration update waiting to be deployed.
If you need a particular Railway configuration file, build-plan output, requirements file, deployment metadata, or command output to diagnose this, tell us exactly what you need and we'll provide it.
20 days ago
I would isolate this without changing the currently healthy production deployment.
The important distinction here is that your failure occurs before any normal Railpack install/build command output. Railway documents Railpack as the layer that analyses the source, installs dependencies, runs the build, and packages the image. If BUILD_IMAGE dies while the only log line is still “scheduling build on Metal builder”, you have not yet reached the part where a Python dependency or application build command would normally produce an error.
A safe A/B test would be to create a temporary service with no production domain, pointing at the exact known-good commit, and reproduce the build there. Do not replace or redeploy the currently healthy production service.
For that temporary service I would make the test as deterministic as possible:
- Pin the same Railpack version explicitly rather than relying on the current default. Railway supports both build.railpackVersion in railway.json and the RAILPACK_VERSION configuration variable.
- Set NO_CACHE=1. Railway documents this as the supported way to disable build-layer caching. This will not fix an unavailable builder, but it removes persistent build cache state as a variable.
- Trigger one build with railway up from the checked-out known-good commit if possible. Railway documents that railway up always uploads and builds the source, so this is also useful for separating GitHub source-trigger behaviour from the actual image-build path.
The resulting matrix is useful:
- Original service fails + temporary service succeeds from identical source: strongly points to service-specific build/cache/config state rather than application code.
- Both services fail before any Railpack build commands appear: strongly points toward the builder/build infrastructure path.
- railway up succeeds while GitHub-triggered deployment fails: investigate source-trigger/config capture rather than Python.
- NO_CACHE=1 changes the result: persistent layer-cache state is implicated.
- A normal Railpack command finally appears and then fails: only at that point does an application/dependency diagnosis become meaningful.
Also compare the Deployment Details configuration from the known-good deployment and a failed deployment. Railway’s Config-as-Code docs state that the deployment details show the settings that deployment actually went out with. Compare builder, Railpack version, root directory, config file, build command and relevant variables rather than assuming the service’s current Settings page exactly matches the historical successful deployment.
Given that production is healthy, I would not delete the service, regenerate domains, change its start command, or touch the running deployment. A disposable service using the known-good commit gives you a much cleaner test and preserves the existing production image while you determine whether the failure follows the source or the Railway service/build path.