Skip to content
Incident 17 Feb ~01:15 UTC

9 months ago

I'm not seeing any discussions related to this in the Discord..

Around 1-2 hours ago, I was notified of 2 things:

  • Hostname/IP does not match certificate's altnames: Host: unproxied-custom-domain-here. is not in the cert's altnames: DNS:*.up.railway.app

  • HTTP 520 responses for Cloudflare proxied custom domains

Looking further, there was also a spike in latency

This affected all of my services, across multiple projects, and all of the deployments were healthy w/o redeploys

The status checker I'm using here is hosted in AMS

Incident started around 01:15 UTC and ended around 01:25 UTC

Screenshot_20260217-045625.png

Screenshot_20260217-045701.png

Screenshot_20260217-045719.png

Solved

40 Replies

9 months ago

What do your http logs say?


9 months ago

Were all green, let me check again


9 months ago

Scrolling through the HTTP logs on a service that got the certificate issue, it's all 200s


9 months ago

Link


9 months ago

noticeable gap (utc+2 tzs)

Screenshot_20260217-050507.png


9 months ago

linked, should be this one


9 months ago

To clearly clarify, this is no longer an issue


9 months ago

Screenshot_20260217-050657.png


9 months ago

I'm not seeing anything on our end here.


9 months ago

in case missed


9 months ago

Yes, I've looked at our monitoring for the time you gave.


9 months ago

another example, cloudflare proxied

Screenshot_20260217-051204.png

Screenshot_20260217-052048.png


9 months ago

Not seeing anything on our end for that either.


9 months ago

I mean, could it be an observability issue? I don't think Cloudflare lies about 520s


9 months ago

I am honestly not sure.


9 months ago

Not a new issue either I believe

Screenshot_20260217-053634.png


9 months ago

I'm also not seeing other users report this πŸ™


9 months ago

"maybe I'm just not like other users"


9 months ago

think about the times where I called out an outage first (at least in the discord)!!


9 months ago

Haha I'm not sure what you want me to say to that.


9 months ago

didn't expect you to


9 months ago

I'm fine with leaving it at this, no SLA or critical services on my end


9 months ago

may be worth looking into (deeply) though..


9 months ago

I've looked, I don't see anything from any of our monitors.


9 months ago

what about now!


haayhappen

9 months ago

I'm also observing this (second time today a couple minutes ago) - first time was 8:20 UTC


mai1015

9 months ago

image.png


mai1015

9 months ago

wow so serious the server down


9 months ago

^


futsy

9 months ago

Also seeing this as well


duxsec

9 months ago

Interested to see what this caused.


futsy

9 months ago

I've been seeing similar behaviour just now.

UTC timeline of the recent flapping I observed (triggered by my monitoring system; which is also monitoring services outside of Railway as well and only Railway was impacted:

08:21:35 β€” DOWN

08:22:56 β€” UP

08:24:58 β€” DOWN

08:26:18 β€” UP

09:40:46 β€” DOWN

09:44:44 β€” UP

09:58:27 β€” DOWN

09:59:47 β€” UP

10:01:53 β€” DOWN

During this window I did some diagnostics:

DNS resolution was normal and stable (single IPv4 A-record; no IPv6).

TCP connect to port 443 succeeded immediately.

The failure happened during TLS negotiation: curl sent the TLS ClientHello, then timed out waiting for the server’s ServerHello (connection established, but the TLS handshake response didn’t come back).

Shortly after, the alert cleared by itself. I re-ran the same diagnostics and TLS completed normally (TLS 1.3), ALPN negotiated HTTP/2, and the health request returned 200.

Note I am NOT behind CF proxy.


pepijn

9 months ago

Same here


9 months ago

"Same" and "this" meaning you all have seen a cert for a service domain instead of a custom domain?


Railway
BOT

9 months ago

Hello!

We've escalated your issue to our engineering team.

We aim to provide an update within 1 business day.

Please reply to this thread if you have any questions!

Status changed to Awaiting User Response Railway β€’ 9 months ago


9 months ago

Happy? πŸ˜†

image.png


9 months ago

I win


9 months ago

<:proud:800594749714071553>


9 months ago

Hello,

We have provisioned more capacity in regards to our edge network in the EU-West region. You should no longer see these blips of incorrect certificates.


haayhappen

9 months ago

Thank you.. how can this be caught automatically next time?


Status changed to Awaiting Railway Response Railway β€’ 9 months ago


9 months ago

We are going to alert on the graph I shared above, and long term, we are well underway on a complete rewrite of the network control plane that will outright solve for this, and solve for many other potential issues.


Status changed to Awaiting User Response Railway β€’ 9 months ago


Status changed to Solved brody β€’ 9 months ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...