8 days ago
cd "D:/projects/domino-cubano" — shell resets to C:\Users\kfish, cd every command.
RECON ONLY. No edits, no commits, no pushes.
FOLD RULE: type all output as literal text. No "+N lines" folds, no
summaries. file:line for every finding.
Run each command separately, no && chains.
-
Find how uvicorn is LAUNCHED in production: search for "uvicorn" in
Procfile, railway.json, railway.toml, nixpacks.toml, package.json
scripts, any start/run scripts, and any Python entry point calling
uvicorn.run(). Quote the exact launch line(s) with file:line.
-
Report whether any of these appear anywhere in the repo:
ws_per_message_deflate, ws-per-message-deflate, permessage,
per_message_deflate. If zero hits, state "zero hits — uvicorn
default (ENABLED) is in effect".
-
Report uvicorn's exact version from the lockfile/requirements, and
which ws protocol impl is installed (websockets vs wsproto —
check requirements/lock for both names).
-
Quote the existing ws_ping settings and where they're set
(expected: ws_ping 20/20 per chat 40's recon — confirm file:line).
STOP after 4. Findings only. If item 2 is zero hits, the follow-up
spec will be a one-flag change to the launch line — do NOT make it
in this run.
3 Replies
8 days ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • 8 days ago
8 days ago
Is there an actual vector to target or is that for us to find... I'd appreciate a repository link if you have one maybe?
8 days ago
Apologies — I pasted the wrong text into this thread. It was an
internal note to my own tooling, not the question. No repository
access is needed. Here is the actual issue:
WebSocket disconnects on all services since ~July 17-18 (US East)
Realtime dominoes app — FastAPI/uvicorn, native WebSockets,
ws_ping 20/20. Ran fine for weeks. Since ~July 17-18 all clients
cycle through constant disconnect/reconnect; recovery takes 2-30+
seconds and clients frequently show "reconnecting" immediately on
page load.
Everything on my side is eliminated:
-
Reproduces on 3 client devices across 3 networks (2 wifi +
cellular).
-
Reproduces after a fresh redeploy of the current build (not
wedged container state).
-
Reproduces after rolling back to a 5-day-old deployment that
ran flawlessly before onset (not my code).
-
Memory flat ~60-80MB over 7 days, no restarts outside deploy
markers (not OOM).
-
Reproduces on a newly added CUSTOM DOMAIN — so this is not the
documented shared-domain proxy termination behaviour.
-
A second, independently deployed copy of the app (deployed
July 10, untouched since) shows identical disconnects.
-
No code changes during the July 15-17 window when it began.
Service: domino-cubano-production.up.railway.app
Region: [FILL IN from Settings → Deploy]
This appears related to the currently-reported US East public
proxy TCP drops (threads dated ~Jul 18 04:00 UTC, Postgres
connections being dropped). Same symptom class, different
protocol. Please check the edge/proxy path for this region.
Fully reproducible on demand — happy to run diagnostics or
provide timestamps.
8 days ago
This is a Railway side issue, not your app. Central Station has an active report of the US East public proxy dropping established TCP connections to Postgres since around Jul 18 04:00 UTC, described as ongoing and not showing up on the status page. Once you're on a custom domain your WebSocket traffic rides that same edge layer, so the symptom class lines up even though the protocol is different.
Independent monitoring backs it up. StatusGator logged an intermittent connectivity incident affecting Outbound Static IPs in US East starting around 6pm on Jul 18, plus a short minor incident shortly after. Railway's own status page shows US East Networking Public uptime already down for July, consistent with degradation starting mid month rather than a one off blip.
There's precedent too. In an earlier thread about recurring 1006 disconnects, Railway staff attributed the disconnects to proxy cycling and restarts on their end and said mitigations were still being worked on.
You've ruled out your code, rollback, memory, and confirmed it on a second untouched deployment plus a fresh custom domain. Stop debugging your app. File a ticket with your region, the Jul 18 04:00 UTC start time, and link the Postgres TCP proxy thread as the same root cause class since it hasn't been formally raised for WebSocket traffic yet. Your ws_ping 20/20 and reconnect logic are already correct. There is no client side fix for a proxy dropping you underneath.