2 months ago
Since approximately 2026-08-18 17:00 UTC, one of my service domains became completely unreachable from the public internet, while the container itself stayed Online and kept serving normally (all scheduled jobs completed on time throughout the outage). The hostname resolves to 69.46.46.41. That IP accepts no TCP connections on port 443 for ANY hostname, tested from four independent networks (two ISPs, a third-party HTTP proxy which returns Cloudflare 522, and a cloud fetch service which returns ECONNREFUSED). It was still dead as of 2026-08-19 16:09 UTC. By contrast, 69.46.46.37 and 69.46.46.79 respond normally to a direct request with HTTP 404 "Application not found", so those edge nodes are healthy and only .41 looks broken. What I tried, in order: several redeploys and a service restart (all completed SUCCESS, edge still unreachable); then deleting the service domain and generating a new one, which landed on .37 and worked instantly, confirming the app, the target port and the deployment were fine the whole time; then renaming that domain back to my original name, at which point the UI showed the name as "Available" and the rename succeeded, but it immediately re-pinned to 69.46.46.41 and went unreachable again. That last step is the interesting one: it suggests the hostname-to-edge-node mapping is deterministic, so my original hostname stays unusable for as long as .41 is down. I have since moved to a different generated hostname (which landed on .79) and I am back online, so this is no longer urgent for me. Two asks. First, can 69.46.46.41 be investigated and repaired or depooled? If the mapping is deterministic, every other service whose hostname maps to that node is presumably also hard-down, and those owners may not realise the cause is on the edge rather than in their own app. Second, once it is healthy I would like to reclaim my original hostname, which is currently unassigned and shows as available. Happy to share project and service IDs and the exact hostname privately if that helps someone look it up.
1 Replies
2 months ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 2 months ago
17 days ago
The deterministic mapping detail is the part matters here. Since a freshly generated domain on .37 and worked instantly, it does sound like a per-node pinning table rather than round robin DNS, so a depool of .41 would probably free those hostnames even before the box itself is fixed.
You could try to delete the custom domain entirely, wait a while for any cache to expire, then re-add it. See if the pin gets rebuilt on re-add or persists in the edge routing config. Also, is 69.46.46.41 still blackholing today or did it recover on its own? A one month old outage on a single edge node with nobody else reporting would be surprising, so if other people with hostnames on .41 commented here it will get picked up faster. Otherwise this needs a staff member, and their Discord is the fastest route for that, post the IP and the timeline there and ask for it to be depooled.