Additional info to help investigate:

Region: ams

Affected projects (both experiencing the SAME simultaneous 1006 disconnects):

1. renewed-growth (mygame.az / kiss.az) - Project ID: d3ab71ea-1fcf-4a2b-a1db-33a23d90e92e, Service ID: b4afa549-fbbd-4170-a29e-1a3230fb5b91
2. calm-empathy (ureyimsen.com) - Project ID: 1565ccb0-d461-4901-a23f-93ca54fd46c1, Service ID: 9b06a761-16a1-4cb7-b40b-c77f3fd68211

Confirmed server-side facts (via logs on both projects):

• No process restart at time of incident
• No uncaughtException / crash
• No deployment triggered at incident time
• In-memory game state (chat history, room state) survives the disconnects
• HTTP and new WebSocket connections work fine immediately after

Client-side observation:

• Browser reports: code=1006, wasClean=false, reason=''
• Multiple UNRELATED users (different accounts, different rooms) disconnect within the same few seconds
• This happens on the custom domain (mygame.az/kiss.az)

Incident times observed tonight (please let me know what timezone reference you need - these are UTC+4):

• ~19:49, 19:51, 19:54-19:55, 20:19-20:20

The fact that this occurs across TWO separate, unrelated Railway projects in the same region at overlapping times strongly suggests an edge-proxy/routing issue in ams rather than an application-level bug. Could you check for edge proxy connection resets, WebSocket upgrade/routing logs, or any infrastructure events in AMS around these timestamps?

Happy to run an A/B test comparing the custom domain vs the Railway-generated *.up.railway.app domain simultaneously if that would help isolate whether this is domain-routing specific.

Since all users drop simultaneously with close code 1006 while your Node.js process stays running without restarting (in-memory state/rooms persist), this is happening at the ingress/proxy layer.

Notice that your drops occurred in rapid succession (19:49, 19:51, 19:54–19:55 — roughly every 2–3 minutes). Here are the 4 exact root causes on Railway that trigger this behavior and how to resolve them:

1. The #1 Cause: Cloudflare "Orange Cloud" (Double Proxy) on Custom Domains

You are serving traffic through custom domains (kiss.az / mygame.az).

• Railway uses Cloudflare for SaaS on its edge to route custom domains. If your DNS provider (e.g., Cloudflare DNS) has the Orange Cloud (Proxied) enabled on your CNAME pointing to Railway, you are running an Orange-to-Orange (O2O) double-proxy setup.
• In this configuration, whenever Cloudflare cycles edge worker connections, rate-limits, or rebalances edge routes, it immediately sends a TCP FIN/RST to all open WebSocket tunnels. The browser receives no WebSocket close handshake and synthesizes code 1006.
• Fix: In your DNS management for kiss.az and mygame.az, change the CNAME record from Proxied (Orange Cloud) to DNS Only (Grey Cloud). Railway handles the SSL certificate and edge termination directly.

1. Railway Healthcheck Timeout Resetting Ingress Connections

Check Service Settings > Healthcheck in your Railway dashboard:

• If you have a healthcheck path (like / or /health) with a strict timeout (e.g. 5–10s):
• When players join or real-time game broadcasts fire, Node.js (single-threaded) can experience a momentary event loop freeze (e.g. JSON serialization, array iterations across rooms).
• If your /health endpoint fails to respond before the timeout, Railway temporarily flags the upstream container as failing/unhealthy and resets all incoming proxy connections, even if it hasn't reached the restart threshold to reboot the container.

Fix:

1. Increase the Healthcheck Timeout to 30s–45s or disable it temporarily to test.

2. Ensure your healthcheck endpoint is ultra-lightweight and returns 200 OK immediately without database or deep state queries.

3. Check for Multi-Replicas or Autoscale

Check Service Settings > Replicas:

If your service is set to more than 1 replica (or autoscaling is enabled), Railway load-balances requests across instances.

Because your game rooms live in Node.js memory (not in an external Redis store), any instance scaling or connection shifting by Railway will drop all clients connected to that node.

Fix: Ensure replicas are locked to 1.

1. Heartbeat (Ping/Pong) Mismatch

If you do not have an active application-level heartbeat, edge proxies and mobile carrier NAT tables silently prune "idle" TCP entries:

• Ensure your WebSocket server pings connected clients every 20–25 seconds:

// Example for 'ws'
const interval = setInterval(() => {
  wss.clients.forEach((ws) => {
    if (ws.isAlive === false) return ws.terminate();
    ws.isAlive = false;
    ws.ping();
  });
}, 25000);

Hello! Thanks for looking into this.

The listed times are in UTC+4. If you need them converted to another time zone to correlate with your logs, that’s no problem.

The events occurred at approximately:

19:49 UTC+4

19:51 UTC+4

Between 19:54 and 19:55 UTC+4

During these events, all WebSocket connections disconnect simultaneously with code 1006, while the Node.js process continues running normally. There are no service restarts, crashes, or unhandled exceptions, and the application state remains intact.

That is why I suspect the issue might be related to the infrastructure/edge proxy or the network path handling the WebSocket connections.

Please check the network path and the events corresponding to those timestamps. If you need further information, logs, or timestamps in a different format or time zone, I can provide them.

Hello,

I have an outstanding invoice dated 2026-09-15 for $28.05 on my account. I'm currently unable to pay the full amount, but I can pay $20 of it right now.

Would it be possible to defer the remaining balance, or set up some kind of partial payment/installment arrangement? Keeping my projects running is very important to me.

Thank you in advance for your understanding.

Account email: terter59095914@gmail.com

Ok raven, no Problem

I am having the exact same issue, but I don't even have a custom domain, and I already have everything set up according to what linkindadaba was suggesting.

The 1006 disconnects don't happen constantly - they happen in bursts at random times throughout the day then stabilize. For instance, today I had the disconnects repeatedly occur for one hour from 9 to 10 am PST then no more after that.

Really hoping to find a solution to this issue. Considering moving to AWS if I can't figure this out :(

That detail about it happening on *.up.railway.app without a custom domain and occurring in 1-hour bursts (like 9–10 AM PST) points to the root cause:

Why this happens on Railway's default domain (*.up.railway.app)

1. HTTP Edge Ingress Proxy Rebalancing / Connection Draining:

   Railway’s default routing uses an HTTP/HTTPS reverse proxy fleet. When you connect via wss://, the connection is established as an HTTP/1.1 Upgrade: websocket tunnel.

   During peak traffic hours (such as 9–10 AM PST, when US West Coast comes online alongside US East and Europe), Railway’s edge ingress routers frequently rebalance, scale, or perform rolling proxy updates. When an ingress node drains or rebalances connections, all upgraded HTTP/1.1 WebSocket tunnels on that proxy node are severed simultaneously via TCP RST/FIN. The browser receives no clean WebSocket close frame and reports 1006.

2. Micro-Healthcheck Failures During Traffic Surges:

   If your service has a Healthcheck Path configured in Railway (Settings > Healthcheck):

   When player activity spikes during peak hours, Node.js single-threaded event loop can experience minor GC pauses or event loop delays. If the healthcheck endpoint exceeds its response timeout, Railway’s edge proxy immediately resets upstream client tunnels while giving the container a retry buffer before rebooting. This causes bursts of disconnects for 30–60 minutes without a container restart.

The Definitive Fix: Switch WebSockets to a Railway TCP Proxy

If you need persistent WebSocket connections that never get dropped by HTTP edge proxy cycling or rolling reloads, bypass the HTTP ingress proxy entirely using Railway's native TCP Proxy:

1. In your Railway dashboard, open your service.

2. Go to Settings > Networking.

3. Under TCP Proxy, click Add TCP Proxy and map it to your WebSocket port (e.g. 3000 or 8080).

4. Railway will provide a direct host and port:

   monorail.proxy.rlwy.net:XXXXX (or similar).

5. In your frontend client, connect directly to this TCP proxy:

   
   // Bypasses the HTTP Edge Proxy completely
   
   const socket = new WebSocket("wss://monorail.proxy.rlwy.net:XXXXX");

Also, regarding your invoice question: Central Station is a public forum and community members can't access account billing. Be sure to reach out to Railway directly at railway.com/help or team@railway.com to get an extension on your bill, and consider editing your post to remove your email address so it doesn't get scraped!

Thanks for the detailed response! I gave the TCP Proxy route a try, but ran into an issue with the code snippet provided.

When attempting to connect to wss://iriguchi.proxy.rlwy.net:XXXX from the client, the browser throws:

net::ERR_SSL_PROTOCOL_ERROR

Woud I have to handle SSL certificates directly inside my Node.js app code to get wss:// working over the TCP proxy, or is there another step I missed?

Ah, that error is almost certainly net::ERR_SSL_PROTOCOL_ERROR

This might be what happenned:

Railway's TCP Proxy is Layer 4 (Raw TCP). Unlike Railway's public HTTP edge (*.up.railway.app), the TCP Proxy does not automatically terminate SSL/TLS for you. If your Node.js backend is running a standard unencrypted server (http.createServer / ws), when the browser attempts a TLS handshake over wss://, the handshake fails because the backend doesn't have an SSL certificate loaded. And if you try plain ws://, your browser blocks it with a Mixed Content error if your site is loaded over HTTPS.

Since you are running on the default *.up.railway.app domain and want a clean setup without managing custom SSL certificates, here is how to fix the root cause of those 9–10 AM bursts directly on your *.up.railway.app domain:

1. Fix Node.js HTTP Server KeepAlive (Most Common Culprit)

In Node.js 18 & 20+, the default server.keepAliveTimeout is only 5 seconds (5000ms). Railway’s edge proxy expects keepalive connections to stay open for at least 60 seconds. When traffic picks up, this mismatch causes Node to silently tear down the underlying socket:

Add this to your Node server entrypoint:

const server = http.createServer(app);

// Must be higher than Railway's proxy timeout (60s)
server.keepAliveTimeout = 65000; 
server.headersTimeout = 66000; 

// Ensure connection upgrade timeout isn't tripping
server.requestTimeout = 0;

I am having the same problem:https://station.railway.com/questions/long-lived-websocket-connections-cut-in-efad9e69

We see exactly this, and it is not your Cloudflare setup: our custom domain is DNS-only, one replica, one region, healthcheck 60 s, socket.io heartbeats already on. The same signature still happens.

Two other threads with the same fault:

• https://station.railway.com/questions/long-lived-websocket-connections-cut-in-efad9e69 (ours, europe-west4)
• https://station.railway.com/questions/websocket-clients-are-reset-on-a-15-seco-962605a3 (us-west2, with the most detailed analysis)

The detail worth checking on your side, because it identifies the cause: when your users get 1006, does your SERVER still think those sockets are open? Ours do. The origin sees nothing at the moment of the cut, and only notices when its next ping goes unanswered; the edge then closes the upstream 15-25 s later. If yours behaves the same way, the connection was dropped between the client and Railway's edge, without the close being passed to your process — which no application setting can fix, and which rules out your game state, your event loop and your DNS.

Our numbers, in case they help you compare: 2026-09-18, 13:07-18:17 UTC, 762 connections cut across 25 channels, in bursts of 30-50 arriving in groups of four about 8-9 minutes apart. The next day (Saturday) had one. Both other threads also report weekday-only windows, which fits the Envoy-draining diagnosis in the us-west2 thread rather than traffic peaks.

One caution about the heartbeat advice. A heartbeat is worth having (it is how the origin notices at all), but it does not prevent the cut. On our side the resets land on OUR OWN ping interval: with pingInterval at 10 s, 58% of 765 resets fell within half a second of a 10-second multiple of connection age. The write is what surfaces the dead path, so a shorter interval finds it sooner rather than avoiding it.

If you can post your region, the exact UTC timestamps of a burst, and whether your server-side sockets stayed open, it becomes much harder for this to be read as three unrelated application problems.

Small correction to my earlier reply: shortening our ping interval demonstrably halved our detection time (39.9 s to 25.0 s median), and it did not stop the bursts, but I have no like-for-like data on whether it changes how often they happen.

Hi @m1cdev,

Your telemetry isolating the write-trigger and the 4-burst cadence is the missing puzzle piece that connects your thread, the us-west2 thread, and the AMS thread.

Here is the exact platform architecture mechanism explaining your data:

1. The 4-Burst Cadence (8–9 mins apart): Rolling Ingress Fleet Updates

The pattern of 4 bursts, 8–9 minutes apart, with a 25–35 minute gap during weekday engineering hours (13:00–18:30 UTC) across both europe-west4 and us-west2 is the signature of a rolling Canary / Deployment pipeline across a 4-node/zone ingress proxy fleet:

• Railway's edge ingress operates across 4 cluster nodes/zones. During a rolling deployment or ingress proxy rebalancing:
  • Node 1 enters a Kubernetes/Envoy draining phase (standard 8–9 minute termination grace period).
  • As Node 1 finishes, Node 2 starts draining (8–9 min), followed by Node 3 (8–9 min), and Node 4 (8–9 min).
  • Total cluster drainage cycle: ~35 minutes.
  • The 25–35 minute gap corresponds to the pipeline's canary bake-time before rolling the next component.
• This explains why Saturday had 1 drop vs 762 on Friday: platform deployments do not run on weekends.

2. Why the Server's Ping Triggers the TCP RST

Your test proving that switching to a 10s ping shifted the drops to 10s multiples is brilliant. Here is why the write triggers the reset:

• When an edge proxy pod enters DRAINING state, it is de-registered from upstream load balancers and stops processing routing tables.
• As long as the connection is idle, neither the browser nor your origin server knows the socket is orphaned (TCP half-open state).
• The moment your Node.js server sends a WebSocket Ping frame (0x89), the proxy attempts to write that frame to the downstream client socket.
• Because the ingress pod is actively draining and its route is detached, the downstream write immediately fails with an EPIPE / ECONNRESET. The proxy’s kernel responds by firing an immediate TCP RST to the client (producing code 1006 with no close frame).
• Meanwhile, on the origin side, Envoy does not send an immediate close frame; it waits for its internal 15-second upstream timeout (pingUnansweredMs median 15.0s), after which your engine.io transport closes and Railway's HTTP log finalizes 15–25s later.
• Key takeaway: A shorter heartbeat cannot fix this—as you noted, more frequent writes simply trigger the drop sooner on draining pods.

3. Concrete Recommendation for Long-Lived WebSockets

Because this is tied to the lifecycle of Railway’s public HTTP reverse proxy fleet, any upgraded HTTP/1.1 WebSocket connection passing through the standard HTTP ingress will undergo periodic connection recycling during weekday platform updates.

To achieve true persistence:

The Permanent Solution: Cloudflare Tunnel (cloudflared)

Running a lightweight Cloudflare Tunnel (cloudflared) container inside your Railway project is the gold standard for long-lived WebSocket workloads:

1. Deploy cloudflared alongside your service.
2. Route your custom domain directly through that tunnel to your Socket.io port.

• Why it eliminates this: Cloudflare Tunnel establishes persistent outbound-only QUIC/HTTP2 tunnels directly from your container to Cloudflare’s global Anycast edge.
• It completely bypasses Railway's public HTTP ingress reverse proxy fleet and pod draining cycles. Your connections will never be affected by weekday platform rebalancing or rolling ingress updates.

(Note: While Railway's native TCP Proxy also bypasses the HTTP ingress fleet, it is raw Layer-4 and does not automatically handle SSL termination for browser WSS clients, which often causes ERR_SSL_PROTOCOL_ERROR unless you manage SSL certificates manually. Cloudflare Tunnel handles SSL termination automatically at the edge).

Your telemetry isolating the write-trigger and the 4-burst cadence is the missing puzzle piece that connects your thread, the us-west2 thread, and the AMS thread.

Here is the exact platform architecture mechanism explaining your data:

1. The 4-Burst Cadence (8–9 mins apart): Rolling Ingress Fleet Updates

The pattern of 4 bursts, 8–9 minutes apart, with a 25–35 minute gap during weekday engineering hours (13:00–18:30 UTC) across both europe-west4 and us-west2 is the signature of a rolling Canary / Deployment pipeline across a 4-node/zone ingress proxy fleet:

• Railway's edge ingress operates across 4 cluster nodes/zones. During a rolling deployment or ingress proxy rebalancing:
  • Node 1 enters a Kubernetes/Envoy draining phase (standard 8–9 minute termination grace period).
  • As Node 1 finishes, Node 2 starts draining (8–9 min), followed by Node 3 (8–9 min), and Node 4 (8–9 min).
  • Total cluster drainage cycle: ~35 minutes.
  • The 25–35 minute gap corresponds to the pipeline's canary bake-time before rolling the next component.
• This explains why Saturday had 1 drop vs 762 on Friday: platform deployments do not run on weekends.

2. Why the Server's Ping Triggers the TCP RST

Your test proving that switching to a 10s ping shifted the drops to 10s multiples is brilliant. Here is why the write triggers the reset:

• When an edge proxy pod enters DRAINING state, it is de-registered from upstream load balancers and stops processing routing tables.
• As long as the connection is idle, neither the browser nor your origin server knows the socket is orphaned (TCP half-open state).
• The moment your Node.js server sends a WebSocket Ping frame (0x89), the proxy attempts to write that frame to the downstream client socket.
• Because the ingress pod is actively draining and its route is detached, the downstream write immediately fails with an EPIPE / ECONNRESET. The proxy’s kernel responds by firing an immediate TCP RST to the client (producing code 1006 with no close frame).
• Meanwhile, on the origin side, Envoy does not send an immediate close frame; it waits for its internal 15-second upstream timeout (pingUnansweredMs median 15.0s), after which your engine.io transport closes and Railway's HTTP log finalizes 15–25s later.
• Key takeaway: A shorter heartbeat cannot fix this—as you noted, more frequent writes simply trigger the drop sooner on draining pods.

3. Concrete Recommendation for Long-Lived WebSockets

Because this is tied to the lifecycle of Railway’s public HTTP reverse proxy fleet, any upgraded HTTP/1.1 WebSocket connection passing through the standard HTTP ingress will undergo periodic connection recycling during weekday platform updates.

To achieve true persistence:

The Permanent Solution: Cloudflare Tunnel (cloudflared)

Running a lightweight Cloudflare Tunnel (cloudflared) container inside your Railway project is the gold standard for long-lived WebSocket workloads:

1. Deploy cloudflared alongside your service.
2. Route your custom domain directly through that tunnel to your Socket.io port.

• Why it eliminates this: Cloudflare Tunnel establishes persistent outbound-only QUIC/HTTP2 tunnels directly from your container to Cloudflare’s global Anycast edge.
• It completely bypasses Railway's public HTTP ingress reverse proxy fleet and pod draining cycles. Your connections will never be affected by weekday platform rebalancing or rolling ingress updates.

(Note: While Railway's native TCP Proxy also bypasses the HTTP ingress fleet, it is raw Layer-4 and does not automatically handle SSL termination for browser WSS clients, which often causes ERR_SSL_PROTOCOL_ERROR unless you manage SSL certificates manually. Cloudflare Tunnel handles SSL termination automatically at the edge).