a month ago
Production is DOWN. Custom domains app.rocketmetrix.ru and www.rocketmetrix.ru are unreachable — TCP connects but the TLS handshake stalls right after Client Hello and times out. This started after the July 13 HA Static IP / network migration.
Project: hearty-contentment / production (region: Southeast Asia)
The application itself is healthy: the latest deploy is ACTIVE/successful, and the generated domain jarviss.up.railway.app works perfectly over HTTPS. Only the custom domains are broken.
Full diagnosis I already did:
-
The old CNAME anchor loox9aqy.up.railway.app resolves to 69.46.46.108, which is a DEAD edge (TLS hangs). The generated domain jarviss was migrated to a live edge (69.46.46.51), but the custom-domain anchors were left on dead edges (app -> .108, www -> 69.46.46.73), so both custom domains went down.
-
To recover I removed and re-added app.rocketmetrix.ru. The new anchor is kk698drt.up.railway.app -> 69.46.46.22 (this edge IS live).
-
I updated DNS at my registrar (reg.ru). The authoritative nameservers now return kk698drt.up.railway.app for the app CNAME, and the domain shows GREEN / verified in the Railway dashboard.
-
However TLS still fails: connecting to 69.46.46.22 with SNI=app.rocketmetrix.ru hangs right after Client Hello, while SNI=kk698drt.up.railway.app on the SAME IP (69.46.46.22) completes TLS fine (returns HTTP 404). So the edge has NO certificate loaded for my custom domain's SNI.
This is a certificate provisioning/loading failure on Railway's edge after the migration. Please:
- Confirm the certificates for app.rocketmetrix.ru and www.rocketmetrix.ru were issued.
- Load/provision them on the new edge (69.46.46.22 / kk698drt.up.railway.app). www also needs fixing — its anchor 616o13b1.up.railway.app -> 69.46.46.73 is a dead edge too.
- Verify the TLS handshake succeeds with SNI for both custom domains.
Please prioritize as urgent — this is production and currently fully down for end users.
Pinned Solution
a month ago
I'm able to access your site just fine. You might be experiencing a region/ISP block on traffic into Railway's edge/servers. Unfortunately, your only option is to use a VPN.
You can also try using Cloudflare's DNS, and using Cloudflare's proxy in front of Railway's CNAME, but you'd need to migrate to Cloudflare's NS for that. (https://developers.cloudflare.com/dns/zone-setups/full-setup/setup/)
9 Replies
a month ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • about 1 month ago
a month ago
This cannot be solved by the community — it is a Railway-side edge TLS
certificate provisioning failure that only Railway staff/infra can fix.
Please route this to the Railway team. This is production-down.
Alternatively I will re-open as "Ask Railway Privately".
a month ago
I'm able to access your site just fine. You might be experiencing a region/ISP block on traffic into Railway's edge/servers. Unfortunately, your only option is to use a VPN.
You can also try using Cloudflare's DNS, and using Cloudflare's proxy in front of Railway's CNAME, but you'd need to migrate to Cloudflare's NS for that. (https://developers.cloudflare.com/dns/zone-setups/full-setup/setup/)
a month ago
Project: hearty-contentment / production (Southeast Asia)
Railway is failing to issue SSL/TLS certificates for our custom domains. This is
production-impacting and started after the July 13 HA Static IP / Metal migration.
Details:
-
New custom domain lk.rocketmetrix.ru: added via "+ Custom Domain". DNS is fully
correct — CNAME lk -> pdfftih7.up.railway.app and TXT _railway-verify.lk are both
in place, and the domain shows a GREEN verified checkmark in the dashboard. Yet
after 40+ minutes NO certificate has been issued. The edge (69.46.46.60) serves no
certificate for this SNI, so browsers get an SSL error ("cannot establish a secure
connection"). openssl s_client returns no peer certificate.
-
Earlier today the exact same thing happened with app.rocketmetrix.ru — it never got
a working certificate either, even after removing and re-adding the domain.
-
Your generated domain jarviss.up.railway.app works perfectly (it uses the shared
*.up.railway.app wildcard cert). ONLY our custom-domain certificates fail to provision.
-
This matches other reports on Central Station of domains stuck on "Issuing TLS
certificate" after the Metal migration.
Impact: our production site is unreachable for our customers, who are actively
complaining right now. We are in Russia and cannot load the site even via VPN, because
the certificate is simply not there to establish a secure connection.
Ask: please provision/issue the TLS certificates for lk.rocketmetrix.ru and
app.rocketmetrix.ru on your edge, or tell us exactly what is blocking issuance on your
side (rate limit? validation failure?).
Timeline: if this is not resolved within the next ~2 hours, we will be forced to
migrate off Railway to another provider — we cannot keep our customers waiting any
longer.
a month ago
I'm able to load lk.rocketmetrix.ru. Try accessing the site from an incognito window or a different device.
Attachments
a month ago
Full summary of the issue and where we landed, for the record.
Project: hearty-contentment / production (region: Southeast Asia).
Symptom: our custom domains app.rocketmetrix.ru and www.rocketmetrix.ru
(and the apex rocketmetrix.ru) were completely unreachable from Russia —
across 3 computers and a phone, on both Wi-Fi and mobile, with and without
VPN. The Railway-generated domain jarviss.up.railway.app and a freshly
created subdomain lk.rocketmetrix.ru worked fine.
Diagnosis (verified with many controlled tests):
-
The TLS ClientHello is dropped at the network level in Russia for the
specific hostnames app/www/rocketmetrix.ru, regardless of destination IP.
Same Cloudflare/Railway IP: SNI=lk.rocketmetrix.ru completes in <1s,
SNI=app.rocketmetrix.ru times out at ~6-8s. Tested against Railway edges,
Cloudflare IPs, and even unrelated Russian servers — same result. So it is
Russia-side, name-based (SNI) filtering of those established hostnames, not
a config/DNS/server issue on our side. The domain is clean on reputation
blocklists (URLVoid 0 detections); it appears to be collateral filtering.
-
Separately, Railway was not issuing TLS certificates for newly added custom
domains (lk and re-added app both sat with green/verified DNS but no cert
for 40+ min). This looks related to the Metal migration — matches other
reports on Central Station of domains stuck on "Issuing TLS certificate".
What we tried, per your suggestion: moved the domain's nameservers to
Cloudflare and proxied the records (orange cloud). This removed the hard
block — app/www now load from Russia — BUT only intermittently. Russian ISPs
throttle parts of Cloudflare's IP ranges, so each request hits a random
Cloudflare IP and succeeds or fails at random. Heavy pages fail more often
(connection reset mid-transfer). So Cloudflare took us from "fully blocked"
to "flaky", but not to reliable.
Conclusion: the root cause is Russia-side blocking/throttling of foreign
hosting reachability for these hostnames, which is outside Railway's control.
For reliable access for our Russian customers we are migrating to a
Russia-based host (domestic IP, traffic stays in-country).
Two questions if you have any input:
-
Is the custom-domain TLS certificate issuance (stuck after the Metal
migration) a known issue, and is there anything to unstick it?
-
Do you have any guidance for serving Russian end-users reliably from
Railway, or is Cloudflare-in-front the only lever on your side?
Thanks for the help so far.
kizhay
Full summary of the issue and where we landed, for the record. Project: hearty-contentment / production (region: Southeast Asia). Symptom: our custom domains app.rocketmetrix.ru and www.rocketmetrix.ru (and the apex rocketmetrix.ru) were completely unreachable from Russia — across 3 computers and a phone, on both Wi-Fi and mobile, with and without VPN. The Railway-generated domain jarviss.up.railway.app and a freshly created subdomain lk.rocketmetrix.ru worked fine. Diagnosis (verified with many controlled tests): - The TLS ClientHello is dropped at the network level in Russia for the specific hostnames app/www/rocketmetrix.ru, regardless of destination IP. Same Cloudflare/Railway IP: SNI=lk.rocketmetrix.ru completes in <1s, SNI=app.rocketmetrix.ru times out at ~6-8s. Tested against Railway edges, Cloudflare IPs, and even unrelated Russian servers — same result. So it is Russia-side, name-based (SNI) filtering of those established hostnames, not a config/DNS/server issue on our side. The domain is clean on reputation blocklists (URLVoid 0 detections); it appears to be collateral filtering. - Separately, Railway was not issuing TLS certificates for newly added custom domains (lk and re-added app both sat with green/verified DNS but no cert for 40+ min). This looks related to the Metal migration — matches other reports on Central Station of domains stuck on "Issuing TLS certificate". What we tried, per your suggestion: moved the domain's nameservers to Cloudflare and proxied the records (orange cloud). This removed the hard block — app/www now load from Russia — BUT only intermittently. Russian ISPs throttle parts of Cloudflare's IP ranges, so each request hits a random Cloudflare IP and succeeds or fails at random. Heavy pages fail more often (connection reset mid-transfer). So Cloudflare took us from "fully blocked" to "flaky", but not to reliable. Conclusion: the root cause is Russia-side blocking/throttling of foreign hosting reachability for these hostnames, which is outside Railway's control. For reliable access for our Russian customers we are migrating to a Russia-based host (domestic IP, traffic stays in-country). Two questions if you have any input: 1. Is the custom-domain TLS certificate issuance (stuck after the Metal migration) a known issue, and is there anything to unstick it? 2. Do you have any guidance for serving Russian end-users reliably from Railway, or is Cloudflare-in-front the only lever on your side? Thanks for the help so far.
a month ago
Railway's edge router doesnt have your SSL certificate loaded.
Go to your Railway dashboard, delete the custom domain, wait 2–3 minutes, and re-add it. You can also try using Cloudflare with a Custom SNI/Proxy Setup
a month ago
Full status update.
Project: hearty-contentment / production (region: Southeast Asia).
The custom domains app.rocketmetrix.ru and www.rocketmetrix.ru are unreachable
for our users. What works and what doesn't:
- lk.rocketmetrix.ru → WORKS (verified earlier, serves a certificate, loads fine).
- jarviss.up.railway.app → WORKS.
- app.rocketmetrix.ru / www.rocketmetrix.ru → FAIL, even from outside Russia via VPN.
Everything is on the SAME service. The only difference is that lk was verified
BEFORE the certificate problem started, and app/www were re-added AFTER.
Following your earlier advice, I deleted and re-added app.rocketmetrix.ru and
www.rocketmetrix.ru, added the new CNAME targets (58edi119 / db3gl94q) and the
_railway-verify TXT records. Both domains now show GREEN / verified in Networking.
BUT the TLS certificate is still never issued for them:
-
They fail even via VPN (from outside Russia) with an SSL / secure-connection
error — so this is NOT a regional block, it is a missing certificate.
-
openssl s_client to the new edge (69.46.46.106, SNI=app.rocketmetrix.ru) returns
NO peer certificate.
-
I also tried putting Cloudflare in front (Cloudflare's own cert): lk works through
it, but app/www still fail to serve, consistent with the cert never provisioning
on Railway's edge.
So: the edge router verifies the domain (green) but the certificate never
provisions. This matches the "certificate stuck after Metal migration" reports on
Central Station. Only the newly (re-)added custom domains are affected; lk (verified
earlier) is fine.
Please have your infrastructure team provision the TLS certificates for
app.rocketmetrix.ru and www.rocketmetrix.ru on the edge, or tell us exactly what is
blocking issuance (rate limit? a stuck ACME order for these hostnames?).
This is production and customers are affected. Thank you.
kizhay
Full status update. Project: hearty-contentment / production (region: Southeast Asia). The custom domains app.rocketmetrix.ru and www.rocketmetrix.ru are unreachable for our users. What works and what doesn't: - lk.rocketmetrix.ru → WORKS (verified earlier, serves a certificate, loads fine). - jarviss.up.railway.app → WORKS. - app.rocketmetrix.ru / www.rocketmetrix.ru → FAIL, even from outside Russia via VPN. Everything is on the SAME service. The only difference is that lk was verified BEFORE the certificate problem started, and app/www were re-added AFTER. Following your earlier advice, I deleted and re-added app.rocketmetrix.ru and www.rocketmetrix.ru, added the new CNAME targets (58edi119 / db3gl94q) and the _railway-verify TXT records. Both domains now show GREEN / verified in Networking. BUT the TLS certificate is still never issued for them: - They fail even via VPN (from outside Russia) with an SSL / secure-connection error — so this is NOT a regional block, it is a missing certificate. - openssl s_client to the new edge (69.46.46.106, SNI=app.rocketmetrix.ru) returns NO peer certificate. - I also tried putting Cloudflare in front (Cloudflare's own cert): lk works through it, but app/www still fail to serve, consistent with the cert never provisioning on Railway's edge. So: the edge router verifies the domain (green) but the certificate never provisions. This matches the "certificate stuck after Metal migration" reports on Central Station. Only the newly (re-)added custom domains are affected; lk (verified earlier) is fine. Please have your infrastructure team provision the TLS certificates for app.rocketmetrix.ru and www.rocketmetrix.ru on the edge, or tell us exactly what is blocking issuance (rate limit? a stuck ACME order for these hostnames?). This is production and customers are affected. Thank you.
a month ago
It might take a few hours maybe even a day for railway to verify domains and TLS certificate for your domain
kizhay
Full status update. Project: hearty-contentment / production (region: Southeast Asia). The custom domains app.rocketmetrix.ru and www.rocketmetrix.ru are unreachable for our users. What works and what doesn't: - lk.rocketmetrix.ru → WORKS (verified earlier, serves a certificate, loads fine). - jarviss.up.railway.app → WORKS. - app.rocketmetrix.ru / www.rocketmetrix.ru → FAIL, even from outside Russia via VPN. Everything is on the SAME service. The only difference is that lk was verified BEFORE the certificate problem started, and app/www were re-added AFTER. Following your earlier advice, I deleted and re-added app.rocketmetrix.ru and www.rocketmetrix.ru, added the new CNAME targets (58edi119 / db3gl94q) and the _railway-verify TXT records. Both domains now show GREEN / verified in Networking. BUT the TLS certificate is still never issued for them: - They fail even via VPN (from outside Russia) with an SSL / secure-connection error — so this is NOT a regional block, it is a missing certificate. - openssl s_client to the new edge (69.46.46.106, SNI=app.rocketmetrix.ru) returns NO peer certificate. - I also tried putting Cloudflare in front (Cloudflare's own cert): lk works through it, but app/www still fail to serve, consistent with the cert never provisioning on Railway's edge. So: the edge router verifies the domain (green) but the certificate never provisions. This matches the "certificate stuck after Metal migration" reports on Central Station. Only the newly (re-)added custom domains are affected; lk (verified earlier) is fine. Please have your infrastructure team provision the TLS certificates for app.rocketmetrix.ru and www.rocketmetrix.ru on the edge, or tell us exactly what is blocking issuance (rate limit? a stuck ACME order for these hostnames?). This is production and customers are affected. Thank you.
a month ago
Also I can access your site without VPN. Both domains are working just fine.
You might want to redeploy your service after these adjustments.
Attachments
Status changed to Solved 0x5b62656e5d • 7 days ago