GitHub pushes to
main
are not reliably triggering deployments
polypropicks
HOBBYOP

a month ago

Hi Railway team,

I’m having a production deployment issue with my project.

Project:

  • Project ID: 869012b9-61e0-42d5-8a71-9982945aae5d
  • Environment: production
  • Services: PREMVP and signal-cache-cron
  • Repo: POLYPROPICKS/PREMVP
  • Branch: main

Problem:

GitHub pushes to main are not reliably triggering deployments, and manual “Deploy latest commit” from the Railway UI is also failing.

Latest relevant commit:

  • 9bd6b71 Feed: cache proactive upcoming gap-fill

What I see:

  • git push origin main succeeds.

  • Railway does not automatically start deployment for the latest commit.

  • When I try PREMVP → Deployments → menu → Deploy latest commit, Railway shows:

    “Failed to trigger deployment — Problem processing request”

  • The same issue appears for signal-cache-cron.

  • Reconnecting the repo/branch temporarily triggered one deployment earlier, but it did not fix the issue permanently.

Expected behavior:

  • Every push to main should trigger deployment automatically.
  • “Deploy latest commit” should manually deploy the latest GitHub commit without failing.
  • Both web and cron services should be able to deploy from the latest connected branch.

Could you please inspect the GitHub webhook/autodeploy state and the backend error behind “Problem processing request”?

This is blocking production deployment for both the web service and cron service.

Solved$10 Bounty

8 Replies

Railway
BOT

a month ago

The deployment failures you experienced were caused by a major service disruption that affected build machines, the dashboard, and other infrastructure between 02:25 UTC and 07:57 UTC on May 20. Both your PREMVP and signal-cache-cron services have since deployed successfully at around 08:36 UTC. If pushes to main are still not triggering deployments, please let us know.


Status changed to Awaiting User Response Railway about 1 month ago


Railway

The deployment failures you experienced were caused by a [major service disruption](https://status.railway.com/incident/I23M92U0) that affected build machines, the dashboard, and other infrastructure between 02:25 UTC and 07:57 UTC on May 20. Both your PREMVP and signal-cache-cron services have since deployed successfully at around 08:36 UTC. If pushes to main are still not triggering deployments, please let us know.

polypropicks
HOBBYOP

a month ago

This did not solve my issue. I still need human help.

The bot response says both PREMVP and signal-cache-cron deployed successfully around 08:36 UTC, but production is still not restored.

Current production state:

  • https://polypropicks.com/ is not serving the app correctly.
  • /api/feed/landing-cards?limit=1 is not returning the expected Next.js API response.
  • The active PREMVP deployment is a railway up deployment, not a clean GitHub deployment.
  • Previous attempts produced Caddy/static runtime behavior and then 502/404 responses.
  • Redeploying an older known-good GitHub deployment is blocked with “Deploys have been paused temporarily”.
  • Manual “Deploy latest commit” also failed earlier with “Problem processing request”.

So the issue is not simply “deployments succeeded”. The current deployment state is broken and production is down.

What I need urgently:

  1. Please manually unpause/repair deployment capability for this project/workspace.
  2. Please restore/redeploy the last known-good GitHub deployment for PREMVP, or force deploy the latest GitHub commit from main.
  3. Please confirm why Railway/Railpack is producing a Caddy/static runtime for a Next.js app instead of running the Node/Next.js server.
  4. Please clarify whether Hobby workspaces are still being paused during the current build incident, and whether upgrading to Pro is currently the only way to get deployments processed.

This is production-blocking. Please escalate to a human engineer.


Status changed to Awaiting Railway Response Railway about 1 month ago


Status changed to Open Railway about 1 month ago


Status changed to Solved polypropicks about 1 month ago


polypropicks
HOBBYOP

a month ago

Update: I tried to restore production by redeploying an older known-good GitHub deployment, not the broken railway up deployment.

That redeploy also failed, but with a Railway build-machine error:

npm ERR! code ENOSPC

npm ERR! syscall write

npm ERR! nospc ENOSPC: no space left on device

Build Failed: failed to solve: process "sh -c apt-get update && apt-get install -y libatomic1" did not complete successfully: exit code: 100

So this is no longer only a GitHub webhook issue or app runtime issue. We cannot restore a known-good deployment because the Railway builder is failing with no disk space.

Current state:

  • Production PREMVP is still down / not correctly restored.

  • Active deployment is a broken railway up deployment.

  • Redeploying a known-good GitHub deployment fails during build with ENOSPC.

  • We need Railway to either:

    1. manually restore/promote the last known-good deployment image without rebuilding, or
    2. provide a clean builder/retry environment with enough disk, or
    3. manually force a clean GitHub-based deployment for PREMVP.

This is production-blocking. Please escalate to a human engineer.


Status changed to Awaiting Railway Response Railway about 1 month ago


polypropicks
HOBBYOP

a month ago

REOPEN REQUEST — this issue is not solved.

The thread was marked solved by mistake. Production is still down / not restored.

I already posted the latest technical evidence below:

  • known-good GitHub redeploy failed inside Railway builder
  • error: ENOSPC / no space left on device
  • active deployment is still broken
  • production cannot be restored from Railway UI

Please reopen this thread and escalate to a human engineer. This is production-blocking.


ve-jo
HOBBY

a month ago

This sounds like the GitHub push event is either not being ingested, or Railway is creating skipped/waiting deployments that are hidden in the default view.

I would check these in order:

  1. In the service Deployments tab, enable/show skipped deployments. Railway docs say pushes can become SKIPPED when Wait for CI is enabled and GitHub Actions fail, or when watch paths are configured and the changed files do not match.
  2. Check Service Settings -> Deploy from GitHub:
    • the trigger branch is exactly main
    • autodeploy is enabled
    • Wait for CI is off unless your GitHub Actions are passing on push
    • watch paths are either empty or include the files changed by your commits
  3. Do not use Redeploy to test this. Railway’s docs say Redeploy uses the exact same code/config as the selected deployment. Use CMD/Ctrl + K -> Deploy Latest Commit to verify Railway can fetch the latest GitHub commit.
  4. If no deployment record appears at all, refresh/rebind the GitHub integration:
    • confirm the Railway GitHub App still has repo access in GitHub installation settings
    • confirm there are no pending GitHub App permission updates
    • disconnect/reconnect the repo in service settings
    • from the project canvas use Add -> GitHub Repository -> Refresh

If all of that passes, the useful data for Railway support is: project, service, environment, repo, branch, exact commit SHA(s), and UTC timestamps for pushes that did not create deployments.


chandrika
EMPLOYEE

a month ago

Your hobby plan builds and deployments have been re-enabled as the platform recovers from a service disruption. You can follow remaining recovery progress here: https://status.railway.com/incident/KVZ1Z8GY

The ENOSPC build error was on our build machines, not your project. With the deploy queue now drained, a fresh GitHub deploy from main should build and run correctly as a Next.js app. The Caddy behavior you're seeing is from the railway up deployment, which uploads your project directory as static files rather than building it through the normal pipeline. You do not need to upgrade to Pro to resolve this.

Please try triggering a deploy from the Railway dashboard (Cmd/Ctrl+K, then "Deploy Latest Commit") or push to main. That should restore your production site.


Status changed to Awaiting User Response Railway about 1 month ago


polypropicks
HOBBYOP

a month ago

Production is now restored after redeploying a clean GitHub-based deployment.

The issue was caused by the incident/builder problems and a bad emergency railway up deployment that produced a Caddy/static runtime. A fresh GitHub-based deployment restored the Next.js app.

Please keep this thread open for now; I will monitor for 12–24 hours before marking solved.


Status changed to Awaiting Railway Response Railway about 1 month ago


Status changed to Awaiting User Response Railway about 1 month ago


polypropicks
HOBBYOP

a month ago

Update: production is restored for now.

We recovered PREMVP by redeploying/rolling back to a clean GitHub-based deployment: Landing: add sports-specific filters with empty teaser.

Current verification:

  • https://polypropicks.com/ returns HTTP 200
  • /api/feed/landing-cards?limit=1 returns HTTP 200
  • main/origin main were reset to the same production-safe commit to avoid accidentally redeploying the broken later runtime/deploy commits.

Please keep this thread open for now. I will monitor for 12–24 hours before marking solved.

Important note for root cause:

railway up produced a Caddy/static runtime and caused 404/502 failures. The known-good GitHub deployment restored the Next.js app.


Status changed to Awaiting Railway Response Railway about 1 month ago


Status changed to Awaiting User Response Railway about 1 month ago


Railway
BOT

a month ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway 27 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...