17 days ago
Hello Railway Support,
Please help recover publication of the latest source for the service selected in this request. The production service remains online, but two Railway Function updates were stuck in APPLYING and the new source has not reached production.
Runtime: Railway Function / Bun 1.4.0.
Timeline (UTC):
- The first commit started on 2026-10-16 at 01:15:55.595.
- The second commit started on 2026-10-16 at 01:54:54.248.
- Both were still APPLYING when checked on 2026-10-16 at 02:08:58.
- A fresh check on 2026-10-17 at 01:31:44 now shows no pending work and nothing staged, but no new deployment has appeared. The latest actual deployment is still the one completed on 2026-10-16 at 01:54:11.
- At 2026-10-17 01:32:22, the public endpoint returned HTTP 200 with exactly the old response content (confirmed by SHA256 comparison), with Cache-Control: no-cache. The service is Online with 1/1 replicas.
What we observed and tried:
- accept_deploy returned "nothing_staged" while the first commit remained APPLYING.
- A separate redeploy completed successfully, but continued serving the old code.
- update_function_source_code with staged:false accepted the complete updated source and created the second APPLYING operation. No new actual deployment followed.
- At the time of the failed update, get_function_source_code returned the latest source in both code and deployedCode, but public HTTP responses still contained the old version.
- The dashboard's Inspect view only said "Changes are being applied" and offered Close. We could not find a recovery/reset control.
Could the Railway team investigate why the accepted source changes did not produce a deployment, confirm the outcome of those commit workflows, and help safely publish the latest saved source?
Please preserve the service, latest saved source, environment variables, and attached persistent volume and database. Please do not delete/recreate the service or detach/delete the volume.
I have omitted internal IDs, source code, credentials, and private logs from this public thread. The affected service is selected in this request; further diagnostics can be provided through a private channel if needed.
Thank you,
Ivan
3 Replies
17 days ago
The service's environment has numbered sets of variables (the NOMACO_DESIGN_ATLAS_*, NOMACO_INTRO_AWAKE_* and NOMACO_INTRO_SLEEP_* groups) that look like asset files split into pieces. Storing files or assets in variables like this is not supported. Once that content gets large enough, the environment can no longer apply changes: the apply looks like it starts, sits on "Changes are being applied", and no new deployment is created. Your Railway Function source itself is fine. Functions store their source in the start command by design, and that part is supported.
Retrying, restaging or pushing the source again won't fix this, and each attempt can make the environment harder to apply. Some of the changes may also have saved even though no deployment followed.
To fix it, move those assets out of variables. You can keep them on the volume already attached to the service, serve them from object storage such as Railway Buckets or any HTTP origin, or deploy them from a repo or image. Then remove the numbered asset variables. After that, the environment can apply changes again and your latest source can deploy. Nothing here requires deleting the service, its other variables, or the volume.
We haven't confirmed exactly what happened to the two specific applies from 2026-10-16, but the variable pattern above is what blocks applies from finishing.
Status changed to Awaiting User Response Railway • 17 days ago
17 days ago
Update for the Railway team — this is still blocked.
We copied all three public assets from the numbered environment variables to the existing attached volume and verified the files. We then staged one patch containing the Function source update to read those files and empty values for the 25 asset variables.
The apply entered APPLYING on 2026-10-17 at 11:56:39 UTC. At 12:40 UTC it was still APPLYING, with no new deployment and no new build/runtime logs. The live service is healthy and still running the 2026-10-16 release.
The variable cleanup itself is pending: we cannot confirm that the effective deployment variables have been cleared. We have not retried or restaged while this apply is running.
Please escalate to a human engineer to inspect lastAppliedError and the commitChanges workflow, unblock the apply safely, and advise how to fully remove the old asset variables.
Please preserve the service, attached volume/database, saved source, and unrelated variables. The affected service is selected in this request; internal IDs are available privately.
Status changed to Awaiting Railway Response Railway • 17 days ago
17 days ago
The apply you started at 11:56 UTC on 2026-10-17 is no longer running but still shows as applying. It contains 26 changes on the service: your new Function source, plus the 25 numbered asset variables set to empty values. Setting them to empty does not remove them. All 25 are still in the service's saved configuration, and some of the patch's writes may have landed even though no deployment followed.
Storing asset files in numbered variables is not supported, and an environment carrying that content can stop staged applies from finishing. The supported step is to delete those 25 variables outright rather than blank them inside a staged patch. Since the assets are now on your volume, the Function source update reading them from there can then be deployed. A stuck apply like this one can later be marked failed automatically. That doesn't remove the variables, so the next apply could hit the same wall while they are still there.
None of this requires touching the service, the volume, the database, the saved source, or NOMACO_DB_PATH and NOMACO_ORIGIN.
Status changed to Awaiting User Response Railway • 17 days ago
6 days ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • 6 days ago