Custom domain TLS not provisioned on new edge after July 13 HA migration (production down)
kizhay
PROOP

a month ago

Production is DOWN. Custom domains app.rocketmetrix.ru and www.rocketmetrix.ru are unreachable — TCP connects but the TLS handshake stalls right after Client Hello and times out. This started after the July 13 HA Static IP / network migration.

Project: hearty-contentment / production (region: Southeast Asia)

The application itself is healthy: the latest deploy is ACTIVE/successful, and the generated domain jarviss.up.railway.app works perfectly over HTTPS. Only the custom domains are broken.

Full diagnosis I already did:

  1. The old CNAME anchor loox9aqy.up.railway.app resolves to 69.46.46.108, which is a DEAD edge (TLS hangs). The generated domain jarviss was migrated to a live edge (69.46.46.51), but the custom-domain anchors were left on dead edges (app -> .108, www -> 69.46.46.73), so both custom domains went down.

  2. To recover I removed and re-added app.rocketmetrix.ru. The new anchor is kk698drt.up.railway.app -> 69.46.46.22 (this edge IS live).

  3. I updated DNS at my registrar (reg.ru). The authoritative nameservers now return kk698drt.up.railway.app for the app CNAME, and the domain shows GREEN / verified in the Railway dashboard.

  4. However TLS still fails: connecting to 69.46.46.22 with SNI=app.rocketmetrix.ru hangs right after Client Hello, while SNI=kk698drt.up.railway.app on the SAME IP (69.46.46.22) completes TLS fine (returns HTTP 404). So the edge has NO certificate loaded for my custom domain's SNI.

This is a certificate provisioning/loading failure on Railway's edge after the migration. Please:

  • Confirm the certificates for app.rocketmetrix.ru and www.rocketmetrix.ru were issued.
  • Load/provision them on the new edge (69.46.46.22 / kk698drt.up.railway.app). www also needs fixing — its anchor 616o13b1.up.railway.app -> 69.46.46.73 is a dead edge too.
  • Verify the TLS handshake succeeds with SNI for both custom domains.

Please prioritize as urgent — this is production and currently fully down for end users.

Solved$20 Bounty

Pinned Solution

I'm able to access your site just fine. You might be experiencing a region/ISP block on traffic into Railway's edge/servers. Unfortunately, your only option is to use a VPN.

You can also try using Cloudflare's DNS, and using Cloudflare's proxy in front of Railway's CNAME, but you'd need to migrate to Cloudflare's NS for that. (https://developers.cloudflare.com/dns/zone-setups/full-setup/setup/)

9 Replies

Railway
BOT

a month ago

This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.

Status changed to Open Railway about 1 month ago


kizhay
PROOP

a month ago

This cannot be solved by the community — it is a Railway-side edge TLS

certificate provisioning failure that only Railway staff/infra can fix.

Please route this to the Railway team. This is production-down.

Alternatively I will re-open as "Ask Railway Privately".


I'm able to access your site just fine. You might be experiencing a region/ISP block on traffic into Railway's edge/servers. Unfortunately, your only option is to use a VPN.

You can also try using Cloudflare's DNS, and using Cloudflare's proxy in front of Railway's CNAME, but you'd need to migrate to Cloudflare's NS for that. (https://developers.cloudflare.com/dns/zone-setups/full-setup/setup/)


kizhay
PROOP

a month ago

Project: hearty-contentment / production (Southeast Asia)

Railway is failing to issue SSL/TLS certificates for our custom domains. This is

production-impacting and started after the July 13 HA Static IP / Metal migration.

Details:

  • New custom domain lk.rocketmetrix.ru: added via "+ Custom Domain". DNS is fully

    correct — CNAME lk -> pdfftih7.up.railway.app and TXT _railway-verify.lk are both

    in place, and the domain shows a GREEN verified checkmark in the dashboard. Yet

    after 40+ minutes NO certificate has been issued. The edge (69.46.46.60) serves no

    certificate for this SNI, so browsers get an SSL error ("cannot establish a secure

    connection"). openssl s_client returns no peer certificate.

  • Earlier today the exact same thing happened with app.rocketmetrix.ru — it never got

    a working certificate either, even after removing and re-adding the domain.

  • Your generated domain jarviss.up.railway.app works perfectly (it uses the shared

    *.up.railway.app wildcard cert). ONLY our custom-domain certificates fail to provision.

  • This matches other reports on Central Station of domains stuck on "Issuing TLS

    certificate" after the Metal migration.

Impact: our production site is unreachable for our customers, who are actively

complaining right now. We are in Russia and cannot load the site even via VPN, because

the certificate is simply not there to establish a secure connection.

Ask: please provision/issue the TLS certificates for lk.rocketmetrix.ru and

app.rocketmetrix.ru on your edge, or tell us exactly what is blocking issuance on your

side (rate limit? validation failure?).

Timeline: if this is not resolved within the next ~2 hours, we will be forced to

migrate off Railway to another provider — we cannot keep our customers waiting any

longer.


I'm able to load lk.rocketmetrix.ru. Try accessing the site from an incognito window or a different device.

20399.png

Attachments


kizhay
PROOP

a month ago

Full summary of the issue and where we landed, for the record.

Project: hearty-contentment / production (region: Southeast Asia).

Symptom: our custom domains app.rocketmetrix.ru and www.rocketmetrix.ru

(and the apex rocketmetrix.ru) were completely unreachable from Russia —

across 3 computers and a phone, on both Wi-Fi and mobile, with and without

VPN. The Railway-generated domain jarviss.up.railway.app and a freshly

created subdomain lk.rocketmetrix.ru worked fine.

Diagnosis (verified with many controlled tests):

  • The TLS ClientHello is dropped at the network level in Russia for the

    specific hostnames app/www/rocketmetrix.ru, regardless of destination IP.

    Same Cloudflare/Railway IP: SNI=lk.rocketmetrix.ru completes in <1s,

    SNI=app.rocketmetrix.ru times out at ~6-8s. Tested against Railway edges,

    Cloudflare IPs, and even unrelated Russian servers — same result. So it is

    Russia-side, name-based (SNI) filtering of those established hostnames, not

    a config/DNS/server issue on our side. The domain is clean on reputation

    blocklists (URLVoid 0 detections); it appears to be collateral filtering.

  • Separately, Railway was not issuing TLS certificates for newly added custom

    domains (lk and re-added app both sat with green/verified DNS but no cert

    for 40+ min). This looks related to the Metal migration — matches other

    reports on Central Station of domains stuck on "Issuing TLS certificate".

What we tried, per your suggestion: moved the domain's nameservers to

Cloudflare and proxied the records (orange cloud). This removed the hard

block — app/www now load from Russia — BUT only intermittently. Russian ISPs

throttle parts of Cloudflare's IP ranges, so each request hits a random

Cloudflare IP and succeeds or fails at random. Heavy pages fail more often

(connection reset mid-transfer). So Cloudflare took us from "fully blocked"

to "flaky", but not to reliable.

Conclusion: the root cause is Russia-side blocking/throttling of foreign

hosting reachability for these hostnames, which is outside Railway's control.

For reliable access for our Russian customers we are migrating to a

Russia-based host (domestic IP, traffic stays in-country).

Two questions if you have any input:

  1. Is the custom-domain TLS certificate issuance (stuck after the Metal

    migration) a known issue, and is there anything to unstick it?

  2. Do you have any guidance for serving Russian end-users reliably from

    Railway, or is Cloudflare-in-front the only lever on your side?

Thanks for the help so far.


kizhay

Full summary of the issue and where we landed, for the record. Project: hearty-contentment / production (region: Southeast Asia). Symptom: our custom domains app.rocketmetrix.ru and www.rocketmetrix.ru (and the apex rocketmetrix.ru) were completely unreachable from Russia — across 3 computers and a phone, on both Wi-Fi and mobile, with and without VPN. The Railway-generated domain jarviss.up.railway.app and a freshly created subdomain lk.rocketmetrix.ru worked fine. Diagnosis (verified with many controlled tests): - The TLS ClientHello is dropped at the network level in Russia for the specific hostnames app/www/rocketmetrix.ru, regardless of destination IP. Same Cloudflare/Railway IP: SNI=lk.rocketmetrix.ru completes in <1s, SNI=app.rocketmetrix.ru times out at ~6-8s. Tested against Railway edges, Cloudflare IPs, and even unrelated Russian servers — same result. So it is Russia-side, name-based (SNI) filtering of those established hostnames, not a config/DNS/server issue on our side. The domain is clean on reputation blocklists (URLVoid 0 detections); it appears to be collateral filtering. - Separately, Railway was not issuing TLS certificates for newly added custom domains (lk and re-added app both sat with green/verified DNS but no cert for 40+ min). This looks related to the Metal migration — matches other reports on Central Station of domains stuck on "Issuing TLS certificate". What we tried, per your suggestion: moved the domain's nameservers to Cloudflare and proxied the records (orange cloud). This removed the hard block — app/www now load from Russia — BUT only intermittently. Russian ISPs throttle parts of Cloudflare's IP ranges, so each request hits a random Cloudflare IP and succeeds or fails at random. Heavy pages fail more often (connection reset mid-transfer). So Cloudflare took us from "fully blocked" to "flaky", but not to reliable. Conclusion: the root cause is Russia-side blocking/throttling of foreign hosting reachability for these hostnames, which is outside Railway's control. For reliable access for our Russian customers we are migrating to a Russia-based host (domestic IP, traffic stays in-country). Two questions if you have any input: 1. Is the custom-domain TLS certificate issuance (stuck after the Metal migration) a known issue, and is there anything to unstick it? 2. Do you have any guidance for serving Russian end-users reliably from Railway, or is Cloudflare-in-front the only lever on your side? Thanks for the help so far.

a month ago

Railway's edge router doesnt have your SSL certificate loaded.

Go to your Railway dashboard, delete the custom domain, wait 2–3 minutes, and re-add it. You can also try using Cloudflare with a Custom SNI/Proxy Setup


kizhay
PROOP

a month ago

Full status update.

Project: hearty-contentment / production (region: Southeast Asia).

The custom domains app.rocketmetrix.ru and www.rocketmetrix.ru are unreachable

for our users. What works and what doesn't:

  • lk.rocketmetrix.ru → WORKS (verified earlier, serves a certificate, loads fine).
  • jarviss.up.railway.app → WORKS.
  • app.rocketmetrix.ru / www.rocketmetrix.ru → FAIL, even from outside Russia via VPN.

Everything is on the SAME service. The only difference is that lk was verified

BEFORE the certificate problem started, and app/www were re-added AFTER.

Following your earlier advice, I deleted and re-added app.rocketmetrix.ru and

www.rocketmetrix.ru, added the new CNAME targets (58edi119 / db3gl94q) and the

_railway-verify TXT records. Both domains now show GREEN / verified in Networking.

BUT the TLS certificate is still never issued for them:

  • They fail even via VPN (from outside Russia) with an SSL / secure-connection

    error — so this is NOT a regional block, it is a missing certificate.

  • openssl s_client to the new edge (69.46.46.106, SNI=app.rocketmetrix.ru) returns

    NO peer certificate.

  • I also tried putting Cloudflare in front (Cloudflare's own cert): lk works through

    it, but app/www still fail to serve, consistent with the cert never provisioning

    on Railway's edge.

So: the edge router verifies the domain (green) but the certificate never

provisions. This matches the "certificate stuck after Metal migration" reports on

Central Station. Only the newly (re-)added custom domains are affected; lk (verified

earlier) is fine.

Please have your infrastructure team provision the TLS certificates for

app.rocketmetrix.ru and www.rocketmetrix.ru on the edge, or tell us exactly what is

blocking issuance (rate limit? a stuck ACME order for these hostnames?).

This is production and customers are affected. Thank you.


kizhay

Full status update. Project: hearty-contentment / production (region: Southeast Asia). The custom domains app.rocketmetrix.ru and www.rocketmetrix.ru are unreachable for our users. What works and what doesn't: - lk.rocketmetrix.ru → WORKS (verified earlier, serves a certificate, loads fine). - jarviss.up.railway.app → WORKS. - app.rocketmetrix.ru / www.rocketmetrix.ru → FAIL, even from outside Russia via VPN. Everything is on the SAME service. The only difference is that lk was verified BEFORE the certificate problem started, and app/www were re-added AFTER. Following your earlier advice, I deleted and re-added app.rocketmetrix.ru and www.rocketmetrix.ru, added the new CNAME targets (58edi119 / db3gl94q) and the _railway-verify TXT records. Both domains now show GREEN / verified in Networking. BUT the TLS certificate is still never issued for them: - They fail even via VPN (from outside Russia) with an SSL / secure-connection error — so this is NOT a regional block, it is a missing certificate. - openssl s_client to the new edge (69.46.46.106, SNI=app.rocketmetrix.ru) returns NO peer certificate. - I also tried putting Cloudflare in front (Cloudflare's own cert): lk works through it, but app/www still fail to serve, consistent with the cert never provisioning on Railway's edge. So: the edge router verifies the domain (green) but the certificate never provisions. This matches the "certificate stuck after Metal migration" reports on Central Station. Only the newly (re-)added custom domains are affected; lk (verified earlier) is fine. Please have your infrastructure team provision the TLS certificates for app.rocketmetrix.ru and www.rocketmetrix.ru on the edge, or tell us exactly what is blocking issuance (rate limit? a stuck ACME order for these hostnames?). This is production and customers are affected. Thank you.

a month ago

It might take a few hours maybe even a day for railway to verify domains and TLS certificate for your domain


kizhay

Full status update. Project: hearty-contentment / production (region: Southeast Asia). The custom domains app.rocketmetrix.ru and www.rocketmetrix.ru are unreachable for our users. What works and what doesn't: - lk.rocketmetrix.ru → WORKS (verified earlier, serves a certificate, loads fine). - jarviss.up.railway.app → WORKS. - app.rocketmetrix.ru / www.rocketmetrix.ru → FAIL, even from outside Russia via VPN. Everything is on the SAME service. The only difference is that lk was verified BEFORE the certificate problem started, and app/www were re-added AFTER. Following your earlier advice, I deleted and re-added app.rocketmetrix.ru and www.rocketmetrix.ru, added the new CNAME targets (58edi119 / db3gl94q) and the _railway-verify TXT records. Both domains now show GREEN / verified in Networking. BUT the TLS certificate is still never issued for them: - They fail even via VPN (from outside Russia) with an SSL / secure-connection error — so this is NOT a regional block, it is a missing certificate. - openssl s_client to the new edge (69.46.46.106, SNI=app.rocketmetrix.ru) returns NO peer certificate. - I also tried putting Cloudflare in front (Cloudflare's own cert): lk works through it, but app/www still fail to serve, consistent with the cert never provisioning on Railway's edge. So: the edge router verifies the domain (green) but the certificate never provisions. This matches the "certificate stuck after Metal migration" reports on Central Station. Only the newly (re-)added custom domains are affected; lk (verified earlier) is fine. Please have your infrastructure team provision the TLS certificates for app.rocketmetrix.ru and www.rocketmetrix.ru on the edge, or tell us exactly what is blocking issuance (rate limit? a stuck ACME order for these hostnames?). This is production and customers are affected. Thank you.

a month ago

Also I can access your site without VPN. Both domains are working just fine.

You might want to redeploy your service after these adjustments.

Screenshot_2026-07-16-18-15-04-36_40deb401b9ffe8e1df2f1cc5ba480b12.jpg

Attachments


Status changed to Solved 0x5b62656e5d 7 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...