The documented deployment dependency ordering gates service startup on other services' reference variables being ready, but it does not explicitly cover bucket provisioning timing during template deploys. When a service's reference variables (e.g. ${{my-bucket.BUCKET}}) resolve before the bucket finishes provisioning, they bake as empty strings, and restarts do not re-resolve them since the variables were already injected at deploy time. A manual redeploy after the bucket is ready is the correct workaround, as it forces a fresh variable resolution pass. Your suggestions for improving this behavior (blocking on bucket readiness, auto-redeploying, or failing loudly on empty references) are solid feature feedback worth posting to Central Station so the team can track demand.

Follow-up with a deeper diagnosis after more testing. The empty value is not a deploy-time bake: it is cached per-variable at variable CREATION time. Evidence: on an affected project, the bucket exists and issues credentials fine, but ${{bucket.X}} references resolve empty persistently. Re-writing a variable (variableUpsert with the same value) immediately fixes THAT variable only; untouched sibling variables stay empty. So the documented workaround needs correcting: a plain Redeploy does NOT heal it - you must re-save each affected variable, then redeploy. Also tested the deployment-dependencies ordering workaround (service referencing a sleep-and-exit service to delay startup past bucket provisioning): 0/3 improvement, consistent with creation-time caching. Total repro across today: 7 of 8 template deploys affected. Hope this narrows it down - looks like the template workflow creates service variables before bucket registration completes, and the resolver caches the miss.

Correction to our earlier reply: a plain redeploy alone does not fix this, as you demonstrated. The per-variable re-save (then redeploy) workaround you found is the right one for now. The deployment dependency ordering documented here covers service-to-service references, not bucket provisioning, which is why the sleep-and-exit delay had no effect either. I've shared this with the team to look into, but can't promise a timeline on the fix

Hey Hasseeb! Good news, we tracked this down and shipped a change off the back of your report 🙌

It wasn't a provisioning race. Your celld template's service variables point at a bucket namespace that doesn't exist. They reference a UUID left over from an earlier version of the template, while the bucket actually bundled with it is named celld-cells. Nothing on the deployed project matches that namespace, so it renders as an empty string on every deploy, permanently.

That also explains the behavior you mapped out. A redeploy can't heal it because there is nothing to resolve to, and re-saving a variable does fix that one variable because saving from the dashboard rewrites the reference to the live bucket. Your "re-save fixes THAT variable only" finding is what pinned it down, so thank you for the extra round of testing.

To fix it for good: update the bucket references in your template's service variables to use the bucket's name, so s3://${{celld-cells.BUCKET}}, ${{celld-cells.ACCESS_KEY_ID}} and so on, then re-publish the template. New deploys will wire up correctly on first boot. For projects already deployed from the old version, the per-variable re-save you found is still the way to recover them.

We also took your "fail loudly" suggestion! Paulo (on-call this week) shipped a change so that publishing a template whose variable references can't resolve now fails with an error naming the exact variables, instead of quietly handing every deployer a broken project.

Let us know if anything still comes up empty after you re-publish and we'll be happy to dig back in.

Happy building 🚂!

Confirmed on my end. Updated the template's variables to name-based refs (${{celld-cells.*}}) and re-published; a fresh deploy now wires up correctly on first boot, self-seed and all. You were right that my race theory was wrong, the stale UUID explains every symptom I measured. Thanks for digging past the first answer, and glad the fail-loudly publish check shipped. Happy building 🚂