2 months ago
Follow-up to our earlier private ticket (station.railway.com/support/intermittent-silent-response-loss-on-shi-6abb3a12), which was auto-closed as "Solved" after 21 days of inactivity without ever being reviewed by an engineer - the two replies we got were both the automated assistant offering a community bounty, twice, despite us explaining why that wasn't the right fit here.
WHY WE'RE FILING THIS PUBLICLY THIS TIME
While researching this we found station.railway.com/questions/intermittent-522-connection-timed-out-556e8e65 - a very close match (same edge region, same "healthy container, edge just doesn't respond" signature). It was resolved by Railway engineers directly in that public thread (not a paid community bounty) once given exact request-ID / X-Railway-Edge evidence, and multiple independent customers corroborating the same symptom in the same window helped get real traction. One Railway engineer (noahd) said on that thread that a fix had "recently" been rolled out around 2026-06-09. We think we may be looking at a regression of the same underlying edge-layer issue, about 3 months later, and would appreciate someone from the infra/networking team checking internal telemetry for europe-west4-drams3a at the timestamps below - this isn't something community members can diagnose, since it requires access to Railway's own edge logs, but we're posting publicly in case other affected customers see this and corroborate (as happened in the referenced thread).
SERVICE DETAILS
Project: Shifttracker (fb059ffd-d5fe-4898-8d23-dcd63db78a0d)
Service: shifttracker-backend, production (744bcf9e-0439-4147-a488-a5565698b3da), environment ef7214fd-2ca8-4af7-86af-b83a16ab8bb0
Public domain: api.shifttracker.se / shifttracker-backend-production.up.railway.app
Every traced incident has edgeRegion: europe-west4-drams3a. Backend container shows zero restarts/redeploys across all investigated windows - this is not an app crash.
SYMPTOM
Our mobile app (React Native / OkHttp on Android) intermittently gets a generic network error with no response at all, which self-heals on retry. We've ruled out our own client-side connection pooling as the sole cause (shipped a native OkHttp keep-alive tuning fix in app v1.3.0 on 2026-09-01 - failure rate afterward was unchanged), and ruled out device/platform specifics (reproduces on different Android hardware, and in a controlled synthetic test over both WiFi and LTE). Two distinct failure signatures show up in our own server-side HTTP logs:
Signature A - response lost in transit: our server receives the request and returns 200, but the client never sees it, retries, and gets a duplicate 200 about 0.6-0.7s later. Example: POST /shift/entry, first attempt 2026-09-02 06:56:02.868 UTC -> 200 in 80ms (requestId eEueigrrRw-U5SLHs_GTAg), retry 2026-09-02 06:56:03.543 UTC -> 200 in 65ms (requestId qnRpbQKvTSeq8FIZs_GTAg). We confirmed via our own idempotency keys that only one row was ever persisted, so no data corruption - but the first response clearly never reached the client. 7 confirmed instances of this pattern so far, all slower POST calls (74-85ms server-side): 2026-09-01 13:56:58, 18:39:03, 19:19:49, 19:34:25, 19:40:20 UTC; 2026-09-02 06:56:01 and 07:42:29 UTC.
Signature B - request never arrives: no corresponding entry appears anywhere in our server-side HTTP logs for the failed attempt - the request appears to die before ever reaching your edge (the same absent-request-id signal that was the key diagnostic clue in the referenced thread). One confirmed instance: GET /tasks/last-logged, 2026-09-02 05:38:10 UTC.
ONGOING RECURRENCE
This is not a one-off: our client-side logs show a steady background rate of 5-9 affected requests/week since mid-August, with a spike to 81 and 63 in the weeks of 2026-08-31 and 2026-09-07 respectively, and it is still happening as recently as 2026-09-30 07:41 UTC.
Happy to share further request IDs, exact timestamps, or anything else if useful - we have a full log of every incident since mid-August.
Thanks!
7 Replies
2 months ago
This thread has been marked as private.
Status changed to Awaiting Railway Response Railway • about 2 months ago
2 months ago
We can't reach the shifttracker-backend service or its environment from the account this thread is tied to, so we can't look at its edge traffic or request logs from here.
Those resources sit under a different account from the one you're writing from. The request needs to come from the account that owns the Shifttracker project before its HTTP logs and the request IDs you listed can be checked against that service.
Status changed to Awaiting User Response Railway • about 2 months ago
2 months ago
The account posting this thread (classebasse) does have access to this service - it's exactly the one selected from this ticket's own Service picker dropdown when filing it (Shifttracker / shifttracker-backend / production), which only lists services the account can see, so access isn't the issue.
To clarify further: our account has two separate Railway workspaces - a Hobby plan ("classesbasse's projects") and the Pro plan workspace "Classes Grus och Gnejs AB", where Shifttracker actually lives. If your support tooling defaults to one workspace context rather than the workspace the selected service belongs to, that would explain why it can't find it. Could you check whether this ticket's lookup is scoped to the right workspace?
Also noting this thread appears to have been automatically converted to private just now, even though we selected "Ask the Community" when creating it - could that be restored, since the public visibility was intentional?
Status changed to Awaiting Railway Response Railway • about 2 months ago
2 months ago
We haven't established what's causing this yet. The community can help investigate your setup.
That's exactly what the Railway community is good at, so we'd like to open your thread as a community bounty. Railway pays a bounty to the community member who solves it, and threads like this usually get picked up quickly.
Opening it makes this entire thread public, including everything already posted. Nothing becomes public until you decide. Use the buttons below.
- Open to the community - Before you click, take a moment to edit or remove anything you'd rather not share. The thread becomes publicly visible right away.
- Keep it private and close the thread - Nothing becomes public and the thread closes.
Status changed to Awaiting User Response Railway • about 2 months ago
2 months ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • about 2 months ago
2 months ago
Opened to the community as suggested, though to be clear: the open question here isn't something community members can diagnose (it requires Railway's internal edge telemetry for europe-west4-drams3a), so we'd still appreciate an engineer taking a look, same as happened in the referenced precedent thread.
Also, the previous bot reply didn't address the workspace-scoping question from our last message: our account has two Railway workspaces (Hobby "classesbasse's projects" and Pro "Classes Grus och Gnejs AB"), and Shifttracker lives under the Pro one. The service was selectable from this ticket's own Service picker, so the account clearly has access - if your lookup tool defaults to the wrong workspace, that would explain the earlier "can't reach it" response.
a month ago
Update and corrections to the original post.
Still happening. Our client-side logs show the same "Network request failed, retrying" pattern through 2026-10-23 04:06 UTC. Weekly row counts for the weeks starting 2026-09-14 through 2026-10-19 are 6, 9, 19, 8, 6 and 16. Rows come in same-second pairs, so there are roughly half as many distinct incidents. Recent client-log timestamps (UTC): 2026-10-23 04:06:47, 2026-10-22 19:41:00, 2026-10-21 19:47:16, 2026-10-21 04:03:13, 2026-10-21 01:12:41, 2026-10-20 02:55:01, 2026-10-19 04:02:35. We have not yet matched these against Railway HTTP logs, so we don't know which signature each one is.
Impact. Low. The app retries and the API is idempotent, so the retries are almost always absorbed with no duplicate data. We have seen at least one case (2026-09-02 05:38 UTC) where both retries failed and a data load failed visibly. We are asking because the loss is unexplained and we can't tell from our side whether it is the edge, the network path or our client.
Corrections to the original post. (1) We wrote that this reproduces on different Android hardware. That is wrong: both affected devices are the same tablet model (Samsung Galaxy Tab A7). The only other device we tested, a Lenovo tablet, ran a synthetic test of about 730 requests over WiFi and LTE with no network failures. So the client device model is not ruled out. (2) "Request never arrives" is based on one instance where the request is absent from our service's HTTP logs. We have not checked headers or request IDs from the failing client, so this does not establish that the request never reached your edge. (3) Our OkHttp keep-alive change did not visibly change the failure rate in client logs, but we have not verified that the change is present in the shipped build.
What we're asking. Could someone with access to edge telemetry for europe-west4-drams3a check whether connection resets or drops are recorded for shifttracker-backend (production) around the timestamps above and in the original post? A result of "nothing recorded" is also useful, since it would point us at the client.
5 days ago
Update and corrections - this can be closed from our side.
Corrections to my reply of 2026-10-25:
(1) The seven "client-log timestamps" I posted were log upload times, not event times. The real event times (UTC) were: 10-19 03:51:34, 10-20 02:21:42, 10-21 00:26:15, 10-21 02:47:25, 10-21 18:05:50, 10-22 17:37:38 and 18:40:38, 10-23 03:52:34. Apologies if anyone spent time matching the wrong ones.
(2) I wrote that the pattern "persists". After classifying each incident individually, that was wrong. There were two different failure types in our client logs, and I had counted them together:
Type 1 (the one this thread is about): first attempt fails with no response, an immediate retry succeeds. In the cases we traced in early September, the edge had received the request and returned 200, but the client never saw the response. This ran at roughly 10 incidents per day across two devices until 2026-09-11 09:42 UTC, then stopped on both devices at once. Since 2026-09-14 we have seen 4 in total.
Type 2: all three attempts fail within about one second, each in ~15 ms. That is the device having no network at all when it wakes up, so it is client-side and unrelated to Railway. Everything I reported for October was this type.
We made no change on our side between 2026-09-01 and 2026-10-25: no backend deploy and no app release.
One question, only if someone happens to know: did anything change on the europe-west4-drams3a edge between 2026-09-11 and 2026-09-14? It would be useful to know whether the stop was a fix on your side or a coincidence (for example on the mobile carrier's side). If nothing is recorded, that is a fine answer too.
Otherwise this is resolved for us. Thanks.