6 days ago
We run a real-time voice application on this service (project: security_guard, service: guard-ai, environment: production, public domain: guard-ai-production-f027.up.railway.app). Twilio Media Streams connect to us over wss://guard-ai-production-f027.up.railway.app/stream and stay open for the duration of a phone call (~90-110 seconds). These are safety-critical security check-in calls.
Over the last 30 days, live WebSocket connections on this domain have been abnormally terminated mid-call 6 times. Each event kills every WebSocket open at that moment simultaneously, while the application process itself is unaffected:
- 2026-07-16 14:21:32 UTC - 1 connection killed - Twilio error 31924 (stream transport cut)
-
- 2026-07-29 19:45:14 UTC - 1 connection - 31924
-
- 2026-08-01 11:45:32 UTC - 2 connections (both live calls, same second) - 31924
-
- 2026-08-06 11:45:46 UTC - 1 connection - 31903 (WebSocket ended prematurely)
-
- 2026-08-14 11:45:50 UTC - 2 connections (both live calls, within 3 ms) - 31903 - our ws close code 1006 (abnormal, no close frame) on both
Evidence that the cut is between Twilio and the app, not in the app:
- On 2026-08-14 both sockets received close with code 1006 and an empty reason in the same event-loop tick; the Node process (same pid, same container d968641448c9) served Twilio's HTTP status webhooks 30 ms later and continued running normally. No deploy, restart, crash, or resource spike on our side at any of these timestamps (deploy logs available).
-
- Two independent TCP connections (from different Twilio media hosts) dying within 3 ms of each other is consistent with a shared proxy/edge element being restarted or rotated, not with endpoint behavior.
-
- Timing note: three of the events occurred in the same daily window, 11:45:30-11:45:50 UTC (Aug 1, Aug 6, Aug 14). Our calls run at fixed daily slots, so we only observe cuts while calls are live - but this clustering may correlate with something periodic on the edge.
Questions:
- Do your edge/proxy logs show connection resets, config reloads, instance rotations, or maintenance on this domain at the timestamps above - in particular around 11:45:30-50 UTC on Aug 1 / Aug 6 / Aug 14?
- Is there a documented maximum WebSocket lifetime, idle policy, or edge-rotation behavior we should engineer around?
- Is there anything we can configure (region, static egress/ingress, dedicated proxy, plan change) that reduces edge-initiated WS terminations for long-lived connections?
We can provide exact CallSIDs, packet-level timing, and our application logs for any of these events on request.
1 Replies
6 days ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 6 days ago
5 days ago
Could you provide the exact request IDs/connection IDs from your platform, if exposed:
- whether your app sends WebSocket ping/pong or app-level heartbeats during the stream,
- confirmation that all affected connections were on the same public domain and not split across domains,
- whether any incident coincided with TLS renegotiation, cert refresh, or ingress IP change on your side.
One practical engineering assumption to adopt meanwhile: treat the platform edge as not guaranteeing uninterrupted 2-minute WebSocket continuity. For safety-critical calls, the safest architecture is usually fast call recovery rather than assuming zero edge drops - for example, detect stream loss immediately, fail over the call path if possible, and alert on clustered simultaneous disconnects.