a month ago
The garageos-daily-backup service in our GarageOS production environment has never deployed.
On October 5 around 20:32 UTC, applying staged function source and S3 configuration became stuck in APPLYING. Around 21:00 UTC it was still applying, with no deployment created.
On October 6 around 04:21 UTC, the old applying operation had disappeared from pending work, but the service remained Offline with no active deployments. Two corrected credential variables are preserved in a newer STAGED changeset. The running web/API services and PostgreSQL remain online.
Please investigate why the apply cleared without creating a deployment and advise how to safely proceed. Please preserve the newer staged credentials, do not reapply older malformed credential values, and leave running services and PostgreSQL unchanged. We need to run a synthetic storage connectivity test before enabling backups.
No secret values, internal operation identifiers or customer data are included in this public report.
1 Reply
a month ago
The October 5 apply did not stall on the backup service's own change. Applying staged changes carries the whole production environment's configuration, and your web and API services hold large amounts of application code and assets as encoded environment variables, split into numbered chunks and decoded by their start commands at boot. The environment has grown too large to apply, so the apply was cut off before a deployment was created, and the stranded operation later settled on its own. Storing code or assets in variables is not supported, and this will repeat for any apply carrying this environment.
Your newer changeset with the two corrected S3 credentials is still staged and unapplied. Because the earlier apply was cut off after its configuration writes, some values it carried may already be live on the backup service even without a deployment. Applying again, splitting the change, or re-staging will not get past this, and each attempt adds more data.
The way forward is to deploy the app from a connected repository or a published image, keep large files on a volume or fetch them from object storage, then remove the encoded variables and any unused service copies. That means changing the running web and API services, so plan it as its own release before applying the backup changeset and running your storage test.
Status changed to Awaiting User Response Railway • about 2 months ago
a month ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • about 1 month ago