Skip to content
Template Health 36%, but fresh deploys are healthy. What's failing?
derekcheungsa
OP

2 months ago

Hi Railway team,

My template "OpenClaw - AI Assistant with Easy Install and Config" (ID 12bc08fb-5a3f-481a-88ae-3c7ebf7ddf49) shows 36% health, with about 1,066 projects and 770 active.

I deployed it into a fresh project with default values. It came up Online on the first try, and /setup/healthz returned 200 in under 1s with no errors in the logs. So I suspect the failures happen after deploy.

Could you tell me:

  1. How is Template Health calculated, and what counts as a failure?
  2. Can you share a breakdown of failures for this template by cause (build, healthcheck, crash or OOM) and by plan?
  3. Do crashes on Free or Trial plans count against the score?
  4. Is there a way for me to see error reasons from other people's deploys?
Solved$20 Bounty

Pinned Solution

1/3: Any sort of failures count towards a template's health.

2/4: No information will be provided due to privacy reasons.

4 Replies

Railway
BOT

2 months ago

We checked our documentation and it does not cover this yet, so the Railway community is the best place to get a real answer: people who have already worked this out on their own projects and can tell you what actually worked.

So we'd like to open your thread as a community bounty. Railway pays a bounty to the community member who answers it, and threads like this usually get picked up quickly.

Opening it makes this entire thread public, including everything already posted. Nothing becomes public until you decide. Use the buttons below.

  • Open to the community - Before you click, take a moment to edit or remove anything you'd rather not share. The thread becomes publicly visible right away.
  • Keep it private and close the thread - Nothing becomes public and the thread closes.

Status changed to Awaiting User Response Railway • about 2 months ago


Railway
BOT

2 months ago

This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.

Status changed to Open Railway • about 2 months ago


1/3: Any sort of failures count towards a template's health.

2/4: No information will be provided due to privacy reasons.


hdnexus

2 months ago

The moderator answered the policy questions, so here's the practical half — because "any sort of failures count" plus your numbers tells you something useful.

1,066 projects, 770 active. Around 300 are gone. If every failure counts and the metric is lifetime rather than windowed, a template with heavy adoption accumulates things that have nothing to do with template quality: people who deployed to try it and deleted it, free-tier projects that ran out of credit, abandoned side projects. 36% on a template that deploys cleanly may be measuring churn more than correctness.

That said, your own test has a gap worth closing. You deployed with default values and checked that it came up — which tests the first 60 seconds. Every failure mode that matters here happens later:

OOM is the one I'd check first. An AI assistant template on default Hobby resources is a plausible OOM factory — the container comes up fine, serves /setup/healthz in under a second, then gets killed the first time someone sends real work through it. Exit code 137 in a deployment's logs is the signature. If you still have that fresh project, drive actual traffic through it and watch the Metrics tab for memory climbing toward the ceiling rather than plateauing. A template that OOMs under use but passes healthcheck would show exactly the pattern you're seeing: healthy deploys, bad health score.

Missing required config is the second. You deployed with defaults, which presumably means you had whatever API keys the template needs already set, or it starts without them. A user who deploys without a valid key gets a container that starts, passes healthcheck, and then fails on first real request — or crashes. Worth deploying once with the required variables deliberately left blank or wrong, and seeing what happens. If it crashes rather than degrading gracefully, that's a fixable cause of failures you'd never see in your own test.

Then version drift. If the template pulls latest or unpinned dependencies at build time, deploys that worked months ago may be failing now on upstream changes. Your deploy today succeeding doesn't cover that — check whether anything in the template resolves unpinned.

On the metric itself: since the breakdown isn't available, the usable proxy is a template changelog. Pin versions, raise the memory default, make missing-config fail loudly at startup with a clear message rather than silently, then watch whether health moves over the following weeks. Slow feedback, but it's the only loop available when per-deploy causes are private.

The question I'd put back to Railway, narrower than the one that got declined: is template health lifetime or windowed, and does a project that deploys successfully and later crashes count against it? That's not per-user data, so privacy doesn't block it, and the answer changes whether fixing the template can actually move the number or whether you're carrying a year of accumulated churn permanently.


hdnexus

The moderator answered the policy questions, so here's the practical half — because "any sort of failures count" plus your numbers tells you something useful. 1,066 projects, 770 active. Around 300 are gone. If every failure counts and the metric is lifetime rather than windowed, a template with heavy adoption accumulates things that have nothing to do with template quality: people who deployed to try it and deleted it, free-tier projects that ran out of credit, abandoned side projects. 36% on a template that deploys cleanly may be measuring churn more than correctness. That said, your own test has a gap worth closing. You deployed with default values and checked that it came up — which tests the first 60 seconds. Every failure mode that matters here happens later: OOM is the one I'd check first. An AI assistant template on default Hobby resources is a plausible OOM factory — the container comes up fine, serves /setup/healthz in under a second, then gets killed the first time someone sends real work through it. Exit code 137 in a deployment's logs is the signature. If you still have that fresh project, drive actual traffic through it and watch the Metrics tab for memory climbing toward the ceiling rather than plateauing. A template that OOMs under use but passes healthcheck would show exactly the pattern you're seeing: healthy deploys, bad health score. Missing required config is the second. You deployed with defaults, which presumably means you had whatever API keys the template needs already set, or it starts without them. A user who deploys without a valid key gets a container that starts, passes healthcheck, and then fails on first real request — or crashes. Worth deploying once with the required variables deliberately left blank or wrong, and seeing what happens. If it crashes rather than degrading gracefully, that's a fixable cause of failures you'd never see in your own test. Then version drift. If the template pulls latest or unpinned dependencies at build time, deploys that worked months ago may be failing now on upstream changes. Your deploy today succeeding doesn't cover that — check whether anything in the template resolves unpinned. On the metric itself: since the breakdown isn't available, the usable proxy is a template changelog. Pin versions, raise the memory default, make missing-config fail loudly at startup with a clear message rather than silently, then watch whether health moves over the following weeks. Slow feedback, but it's the only loop available when per-deploy causes are private. The question I'd put back to Railway, narrower than the one that got declined: is template health lifetime or windowed, and does a project that deploys successfully and later crashes count against it? That's not per-user data, so privacy doesn't block it, and the answer changes whether fixing the template can actually move the number or whether you're carrying a year of accumulated churn permanently.

derekcheungsa
OP

2 months ago

Great anwser. Thx!


Status changed to Open medim • about 2 months ago


Status changed to Solved 0x5b62656e5d • 18 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...