a month ago
## Summary
After the Apr 26 "Degraded Build Performance" incident (resolved 22:17 UTC), my service Touross stopped receiving GitHub push events from the Fraboli365/Touross repo. GitHub continues to send events correctly — Vercel, configured on the same repo, receives and deploys all of them. Railway does not detect any new commit pushed to main since the incident.
## Project / Service / Environment
- Project: acceptable-love
- Service: Touross
- Environment: production
- Region: us-east4-eqdc4a
## Timeline (UTC)
- Apr 26 ~16:32: GitHub-triggered deploy for PR #24 (commit f558eeb) FAILED at Initialization > Snapshot stage with error "##NOT-AUTHORIZED## repository not authorized". Build stage was already successful at 00:00 before the failure.
- Apr 26 17:10: triggered manual deploy via railway up to recover production. Deploy succeeded (deploy ID dfa2d1b6).
- Apr 26 21:14 — 22:17: Railway global incident "Degraded Build Performance" (per status.railway.com).
- Apr 26 22:30 onwards: pushed several commits to main (19cff27 and a follow-up). Vercel detected and deployed all of them. Railway detected NONE.
- Apr 26 ~22:40: tried Disconnect + Reconnect of Source Repo from Railway dashboard. UI shows repo connected correctly. No effect.
- Apr 26 ~22:50: tried Disconnect + Reconnect of just the branch (main). Reconnecting the same branch was treated as no-op by Railway (no "Apply changes" prompt appeared). No effect.
- Apr 26 ~22:55: triggered Redeploy from latest active deployment via UI 3-dots menu. Succeeded. Then pushed another test commit to main. Still not detected by Railway.
## Verified state of the GitHub App
- "Railway App" is installed in the Fraboli365 GitHub account.
- Repository access: "Only select repositories" → Fraboli365/Touross is in the list.
- Permissions: read/write on actions, administration, checks, code, commit statuses, deployments, pull requests, workflows. No pending approvals, no warnings.
- Vercel works correctly on the same repo and same push events, so GitHub is delivering events normally.
- "Wait for CI" is OFF in the Railway service settings.
## Hypothesis
The service-level webhook subscription state appears to be corrupted post-incident. The dashboard UI shows the connection as healthy, but GitHub events are not being processed by the service. Disconnecting and reconnecting from the UI does not flush this internal state.
## Request
Please investigate the webhook subscription state for service Touross in project acceptable-love. I suspect a backend-side reset of the GitHub event subscription is needed, which the UI cannot trigger.
## Reference deploy IDs
- FAILED Apr 26 16:32 UTC (PR #24): c6ce88bd
- ACTIVE Apr 26 17:10 UTC (manual railway up): dfa2d1b6
- ACTIVE Apr 26 ~22:55 UTC (manual Redeploy via UI): the latest ACTIVE in current state
## Workaround currently in use
Manually triggering deploys via railway up after each merge to main. Would appreciate a fix so automatic deploys resume.
Thanks!
3 Replies
Status changed to Awaiting Railway Response Railway • 26 days ago
a month ago
Update — webhook is working again as of ~22:30 UTC, no manual intervention
required.
Timeline:
- 22:00 UTC: triggered Redeploy via UI 3-dots menu (last attempted action
before opening this thread)
- 22:30 UTC: posted this thread
- 22:30 — 23:25 UTC: gap (~1h, no actions taken)
- 23:25 UTC: merged PR to main from GitHub web. Railway picked up the push
automatically and deployed "via GitHub" successfully. Migration applied
via preDeployCommand without issues.
Possible causes (not confirmed): backlog from the Build Performance incident
clearing with delay, the manual Redeploy taking effect retroactively, or
something fixed silently on Railway's end.
Issue is no longer urgent. Leaving this thread open as reference in case
the diagnostic info is useful for anyone else hitting the same pattern.
Thanks for the platform!
Status changed to Solved fraboli365 • 26 days ago
a month ago
Update 2 — webhook is intermittent, not fully recovered as initially reported.
Evidence after the previous update:
- 2026-04-26 23:25 UTC: merged PR #27 to main from GitHub web. Railway
picked up the push automatically and deployed "via GitHub" successfully.
Migration applied via preDeployCommand. Reported as "recovered" in
previous update.
- 2026-04-27 00:39 UTC (~12 min later): merged PR #28 to main from GitHub
web. Same branch, same repo, similar push event. Railway did NOT pick
up this push. Last deploy in Railway dashboard remained the SHA from
PR #27. Waited 180s, no new deployment appeared in queue/building/active.
Both pushes had identical conditions: same target branch (main), same
GitHub App permissions, same Railway service config, same author. The
only difference is timing and content. PR #27 was a code change
(perf optimization + Prisma migration), PR #28 was docs-only (single .md
file edit).
Pattern looks like message loss or selective filtering somewhere in the
GitHub→Railway event pipeline rather than a complete configuration break.
Possibly a consumer that drops events under certain conditions, or a
rate-limited path triggered intermittently.
Workaround still in use for missed deploys: railway up --service Touross
manually after merge.
Reference deploy IDs:
- PR #27 (auto-deployed via GitHub): commit 6cbe43c9, deployed
~2026-04-26 23:28 UTC
- PR #28 (NOT auto-deployed): commit c3b2c5b7, no Railway entry as of
~2026-04-27 00:42 UTC
Happy to provide more samples if helpful.
Status changed to Awaiting Railway Response Railway • 26 days ago
Status changed to Solved fraboli365 • 26 days ago
25 days ago
Hi, i've looked into the logs and we've received the pushes you mentioned. The commit c3b2c5b7 you mentioned was skipped because of your watch path configurations: the commit did not match any files defined in your watch paths, so no new deployment was created.
I've seen the same from other pushes that you mentioned, which were skipped because of your watch paths.
If you want these commits to trigger a new deploy, you can adjust your watch paths settings in your service.
Status changed to Awaiting User Response Railway • 25 days ago
Status changed to Solved timomeh • 25 days ago