a month ago
Project type: Dockerfile-based Python service
Source: GitHub repo, main branch, auto-deploy enabled
Summary
Auto-deploy-on-push stopped working for one of my services. The
failure is total and silent — no deployment record is created at all
for affected commits (not failed, not skipped, just absent from
history). Manual --from-source redeploys and even a full
disconnect/reconnect of the GitHub source didn't fix it — every
attempt kept redeploying the same stale commit instead of pulling the
current branch tip. Only got unblocked by using railway up to deploy
directly from a local working directory, bypassing GitHub entirely.
Timeline
Pushed a commit to main, followed by 5 more commits. Prior commits
over the preceding two days had deployed reliably, ~10–16 min after
push (normal build time for this service).
Auto-deploy didn't fire for any of the 6 commits. ~18 min past normal
lag, confirmed via railway status --json that production was still
running the last commit before the gap.
Ran railway redeploy --service --from-source -y manually.
Reported SUCCESS, but re-deployed the same stale commit rather than
pulling the branch's actual current tip.
Disconnected/reconnected the branch connection (Settings → Source
→ branch row → Disconnect → Connect Environment to Branch). Deployed
again afterward — no change, still stale.
Disconnected/reconnected the repo-level GitHub source (Settings →
Source → Source Repo → Disconnect → reconnect). Ran another
--from-source redeploy — SUCCESS again, but still deployed the same
stale commit, not the branch's current tip.
Used railway up --ci to deploy directly from a local working
directory (clean tree, HEAD matching the branch's actual tip). This
worked immediately and the correct code is now live.
What I ruled out
"Wait for CI": off
"Skipped Builds": off
Source repo/branch correctly configured, matching what's actually
in GitHub
Auto-deploy toggle: enabled
Deployment history (including skipped deployments shown) has zero
entry of any kind for the 6 missing commits
Also found: this isn't a first-time thing for this project — 8–9
prior instances of unusually long commit-to-deploy gaps (12–66 hours)
over the past two months, though some of those could be ordinary
manual redeploys rather than genuine misses. This incident is the
clearest case — a clean, total, zero-record miss for 6 consecutive
commits, resistant to two separate reconnects plus two explicit
--from-source pulls.
Question for the community
Since both the passive webhook path AND an explicit --from-source
pull (twice, after two separate reconnects) failed to fetch the
branch's actual current commit, this feels like it's on the
build-source side rather than a plain webhook delivery miss — maybe a
stale cached commit reference that doesn't clear on reconnect, or a
GitHub App installation token issue. Has anyone else hit this? I saw
a report of a GitHub OAuth App-token rate-limiting issue causing
silently dropped webhook deliveries platform-wide around late January
this year — wondering if this is a recurrence of something similar, or
a different root cause. Any way to force-clear a cached source
reference, or is filing a direct support ticket the only path once
community troubleshooting is exhausted?
Currently unblocked via railway up, but would like the underlying
auto-deploy path working again rather than staying on a manual-deploy
workflow indefinitely.
7 Replies
a month ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 1 month ago
a month ago
Try Redeploy Latest Commit
Open your service, open the command palette (Ctrl-K) and select "Redeploy Latest Commit"
a month ago
Thanks for the suggestion! Just tried it — no luck unfortunately, but it did give me some useful info.
It redeployed the exact same stale commit we've been stuck on, same as our CLI --from-source attempts. So at least that rules out the CLI as the culprit — both the dashboard button and the CLI are hitting the same stuck reference somehow.
The annoying part: it also wiped out a working deploy we'd gotten live via railway up a bit earlier, and reverted us back to the broken code. So it's not just "doesn't fix it" — triggering a redeploy through any path seems to actually override whatever's currently running and fall back to that same old commit. Had to redeploy manually again to get back to a good state.
a month ago
If you create a new service using the same github repo as the source, does it deployed from the latest commit you expect to be deployed?
a month ago
Spun up a fresh project, same repo/branch, zero history. Railway auto-detected the monorepo and created 6 services at once — every single one resolved "latest commit" to the same stale commit we've been stuck on for hours. Not main's actual tip. Two later failed their deploy step, but only after a successful build, looked like missing env vars on a brand-new service — unrelated.
So: brand-new project, brand-new services, nothing shared with our original setup, and they all got the same wrong commit. Rules out anything specific to our ai-service. Whatever's holding onto that stale reference is upstream — feels like it's stuck at the GitHub App installation level for this repo.
Also tried the dashboard's "Redeploy Latest Commit" button as a different trigger path. Same wrong commit.
a month ago
Following up on the stale-commit issue from earlier in this thread (the
one where every service, even brand-new ones in a fresh test project,
kept resolving "latest commit" to the same wrong commit).
Went ahead and tried the standard fix — fully uninstalled the Railway
GitHub App from GitHub's side (Settings → Applications → Installed
GitHub Apps → Uninstall), then reconnected fresh through Railway's
Source Repo settings.
That made things worse, not better: the reconnect now fails outright
with "Bad credentials" shown right in the Source Repo panel. Tried
it again in a fresh incognito session in case it was a stale
browser/session thing — same result.
So at this point:
Confirmed (via a 6-service diagnostic in a totally separate, brand-new
project) that the stuck reference is at the GitHub App
installation/account level, not tied to any one service
A full uninstall + reinstall of that installation doesn't fix it —
it just breaks the connection entirely with bad credentials
Not a browser/session caching issue (retested incognito)
I don't have a Pro plan so can't open a private ticket — is there
anything else worth trying from my end, or does this need someone on
the Railway team to look at the account/installation directly?
a month ago
I dont think so, there are help channel on their discord but its now archived so.. i think we have to wait until someone from team/mod check this thread
Also, do you moved the repo to an orgs?
a month ago
Turns out the whole "stale commit" thing was my own mistake, not
anything wrong with Railway. Local commits (including the fix I was
trying to deploy) got committed but never actually git push-ed after
an earlier push in the same session — so origin/main on GitHub
genuinely never advanced. Railway was deploying exactly what GitHub
had the whole time; it was working correctly the entire way through.
Sorry for sending you down a path chasing a bug that didn't exist —
and thanks for your patience and suggestions along the way (the
redeploy-latest-commit tip was good troubleshooting, it just couldn't
have worked since the real issue was on my end, not the deploy
trigger).