5 days ago
Incident report — total outage, 2026-08-16
Project: esraabaya (Railway) · Service: ESRA ABAYA (web) · Domain: esraabaya.store Impact: complete outage, HTTP 404 from the edge, ~3h40m Window: 2026-08-16 11:48 → 15:28 (UTC+3) = 08:48 → 12:28 UTC
What our monitoring saw
External uptime monitor, four independent probes, all reporting 404 Not Found:
11:48:38 detected — Ashburn, USA 178.156.189.113
11:48:54 confirmed — Ohio, USA 52.15.147.27
11:49:09 confirmed — Dallas, USA 216.144.248.28
11:49:25 confirmed — N. Virginia, USA 54.87.112.51
11:49:27 alert email sent
15:28:04 incident resolved
What our application data shows
We record one row per visitor session per day, written synchronously inside the request (not queued), so its timestamps are true request times:
Hour (UTC+3) Sessions
09:00 61
10:00 63
11:00 46
12:00 0
13:00 0
14:00 0
15:00 40
16:00 74
Baseline is 60–70 sessions/hour around the clock. Zero requests reached the application for three consecutive hours. Only 3 requests landed in the entire 11:48–15:28 window.
A second signal: our queue worker (separate service, same project) stopped processing jobs at roughly 09:00, about three hours before the web service went down, and drained its backlog between 12:00 and 15:00. Two independent services degraded within the same few hours.
What we ruled out
Hypothesis Evidence against
A deploy of ours broke it Last commit before the window: 00:11. Next: 15:25. No deployment occurred during the outage.
An admin action Our activity log is empty between 10:00 and 16:30.
A removed route Every URL returns 200 today; no routing change shipped that day.
Application error A Laravel error would return 500, not 404 — and would have been logged. Nothing reached the app at all.
The outage ended the moment we pushed an unrelated commit at 15:25, which produced a new deployment at ~15:28. No code in that commit touches routing, domains, or startup. The recovery correlates with a new deployment existing, not with anything the deployment changed.
Our reading
The service appears to have stopped and failed to come back. Our restart policy was ON_FAILURE with maxRetries: 10; once those were exhausted, nothing retried, and the domain served 404 from the edge until a manual deploy created a fresh deployment.
We have already fixed our side of this (see below). What we cannot see is why the container stopped or why its restarts failed — Railway's CLI only retains logs for the current deployment, so that window is gone for us.
What we're asking
Do your host-level logs show what happened to service ESRA ABAYA between 08:48 and 12:28 UTC on 2026-08-16 — an OOM kill, a host migration, a rescheduling failure, or repeated boot failures?
Was there any infrastructure event affecting our project's region or host in that window? Nothing is listed on status.railway.com for 2026-08-16, and we understand isolated issues aren't published there.
Can you see why the restarts did not recover the service, and whether the domain remained mapped throughout?
Our queue worker stalled ~09:00 the same morning. Were both services on the same host?
What we changed on our side (already deployed)
Our own configuration turned a transient failure into a 3h40m outage, and we've addressed that independently of the cause:
Startup no longer gates the web server on database work. The container previously ran six commands chained with && before starting the server (storage:link && migrate --force && db:seed --force && config:cache && route:cache && view:cache && frankenphp). Any one failing meant no server at all. It now waits up to 60s for the database, and starts the server regardless; a failed migration aborts boot only when the database is reachable (a genuine bad deploy, where keeping the previous deployment is correct).
Health check — we're setting healthcheckPath to /up (Laravel built-in, cheap and unthrottled). Our richer /healthz probe — database, cache, queue worker, scheduler, pending migrations — stays for external uptime monitoring.
Restart policy — moving from ON_FAILURE (10 retries, then permanent death) to ALWAYS.
1 Replies
5 days ago
The logs for the web service show a clean, graceful shutdown at 08:43:58 UTC on 2026-08-16, with FrankenPHP stopping normally and the container exiting with code 0. The queue worker's last processed job was at 08:43:59 UTC, confirming both services stopped at the same moment rather than hours apart. Because the exit was clean (code 0, not a crash), the ON_FAILURE restart policy correctly treated it as an intentional stop and did not retry. No host maintenance events or platform incidents were recorded for that window. We cannot determine from the retained logs what delivered the shutdown signal to both services, but the outage duration was a consequence of ON_FAILURE not restarting a code-0 exit, which your switch to ALWAYS directly addresses.
Status changed to Awaiting User Response Railway • 5 days ago