Skip to content
Edge intermittently refuses TCP connections (~12%) — affects both custom domain and *.up.railway.app
amnatteenet
OP

a month ago

Service: rubfung-transcribe (Southeast Asia, 1 replica, Hobby)

Custom domain: rubfung.com

Date observed: 2026-10-23, roughly 17:30-18:40 ICT (UTC+7), ongoing

SYMPTOM

About 12% of new HTTPS connections to our service are never accepted at the TCP

layer. curl reports:

curl: (28) Failed to connect to rubfung.com:443 after 10002 ms

http_code=000 time_connect=0.000000

time_connect is exactly zero on every failure, so the connection never reaches

our container. Successful requests in the same minute return in ~0.05s.

MEASUREMENTS

27 failures out of 230 attempts (11.7%), collected from two independent machines

on two different networks. A later 10-request burst failed 3 times (30%), so the

rate fluctuates considerably minute to minute.

CONTROLS (fired within the same second as each target request)

https://www.cloudflare.com/cdn-cgi/trace 0 failures / 120

https://www.gstatic.com/generate_204 0 failures / 60

WHAT WE HAVE ALREADY RULED OUT

  1. Not our application: the requests never reach it. We also confirmed the

    container is not restarting — our /api/version endpoint reports a constant

    process start time across all successful requests.

  2. Not our network: controls fired in the same second never failed.

  3. Not Cloudflare: our domain is not proxied through Cloudflare. Responses carry

    "Server: railway-hikari".

  4. Not IPv6: the domain has no AAAA record, and forcing IPv4 shows the same rate.

  5. Not specific to our custom domain: the same failures occur against

    rubfung-transcribe-production.up.railway.app.

  6. Not fixed by redeploying: a fresh deployment (new container) still shows it.

IMPACT ON VISITORS

Our site is served over HTTP/2 (verified in a real browser: the navigation entry

and every resource report nextHopProtocol "h2"), so a page load uses a single

multiplexed connection rather than one connection per asset. When that connection

is refused, there is nothing to fall back on and the whole page stalls, not just

one image. At the measured ~12% failure rate that is roughly one page load in

eight arriving as a blank, hanging page for 10-20 seconds.

ALSO SEEN FROM A THIRD NETWORK

A headless Chromium running in a Modal container (completely separate network)

could not finish loading any page of our app, while single-file endpoints on the

same origin (robots.txt, sitemap.xml) loaded instantly. Pages that needed a

connection hung; single files that happened to get a good connection returned at

once.

QUESTION ABOUT YOUR EDGE PROXY

Is our service behind the newer edge proxy, and if so, is there a way to move it

off? We looked for a setting under both service Networking and workspace settings

and found nothing we can change ourselves.

QUESTION

Is there a known issue with edge connection handling in Southeast Asia, or with

our service's edge routing specifically? Your status page currently lists

"Networking - Public" at 100% for October, which does not match what we measure.

CURRENT STATUS - PLEASE READ BEFORE TESTING

On 2026-10-24 at 18:45 ICT we placed the domain behind Cloudflare's proxy. That

removed the user-visible symptom completely: 0 failures in 520 requests across

two machines over the following 12 hours, including an overnight run of 300

consecutive requests with zero failures. The night before, the identical script

with the same parameters over the same hours recorded 29 failures in 300

requests. The only variable that changed between those two nights is whether

traffic reaches your edge directly. Median connect time also dropped from 56ms

to 22ms.

The behaviour at your edge is unchanged - we have only routed our visitors

around it, and your other customers who do not sit behind a proxy are still

exposed to it. Every measurement in this report was taken while the domain was

unproxied, between 2026-10-23 17:30 and 2026-10-24 18:45 ICT. If you test

rubfung.com now you will not reproduce it.

We are happy to run any additional capture you need, and we can re-expose the

domain directly for a window if you want to reproduce it yourselves - just tell

us when. We have a reproducible measurement script and raw logs with timestamps.

$10 Bounty

2 Replies

Railway
BOT

a month ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway • 26 days ago


There isn’t a way to move the traffic away from Railway’s proxies. Seeing as how the issues were resolved after placing the site behind Cloudflare’s proxy, it would seem like this has something to do with your ISP. There also isn’t an issue regarding edge routing.


nilu-mukherjee

a month ago

I checked both hostnames in the report with a limited read-only run on October 26, 2026, 03:55:46–03:56:20 IST (October 25, 22:25:46–22:26:20 UTC), from one Windows network location.

Tests performed

  • System DNS currently returns Cloudflare addresses for rubfung.com. Its 20 fresh IPv4 HTTPS GETs to /robots.txt all returned HTTP 200 with Server: cloudflare.
  • rubfung-transcribe-production.up.railway.app resolved to 69.46.46.114, with no AAAA answer returned. Its 20 fresh IPv4 HTTPS GETs to /robots.txt all returned HTTP 200 with Server: railway-hikari.
  • Three TCP-only connections to 69.46.46.114:443 succeeded. Five interleaved gstatic controls returned HTTP 204.

Each HTTPS request used a separate curl process, no retries, TLS verification enabled, a 5-second connection timeout and a 10-second total timeout. These tested HTTP/1.1, not browser HTTP/2.

I did not reproduce a connection timeout or refusal. There was one slow successful Railway request at 2026-10-25T22:26:11.9837382Z: TCP phase about 1.059 seconds, versus a 41.88 ms median across the 20 samples. It completed with HTTP 200 in 1.170 seconds, request ID IH1tFnKuTeesUv3qAXC71g. Without packet capture, this does not establish retransmission or an edge defect.

The current Cloudflare result is consistent with your mitigation, but it does not retest the original direct custom-domain path. Twenty successful Railway-hostname requests from one location also cannot establish that the historical issue is fixed.

One distinction in the original report matters: curl error 28 indicates a timeout, not proof of an active TCP refusal. time_connect=0 means no completed connect was recorded; it does not identify the failing hop. Cloudflare success and healthy controls to other destinations do not, on their own, establish an ISP cause or a Railway cause.

Proposed next tests — not performed:** during a recurrence, capture time-aligned failures from two independent source networks, with destination IPs, UTC timestamps, curl phase timings and client-side handshake packets. Railway could correlate those exact flows with edge telemetry. With the owner's participation, curl --resolve against the confirmed custom-domain ingress IP would also let you test that SNI/Host route while leaving Cloudflare enabled.


Welcome!

Sign in to your Railway account to join the conversation.

Loading...