Railway <> Hetzner servers cant communicate
mcpizza0
PROOP

2 years ago

We had a deployment running succesfully for the last week

today, the railway instances cant communicate with the servers in Hetzner datacenter, and the otherway around is true too

is there some way we can diagnose this?

feels like railway has blocked the whole ip range

77 Replies

mcpizza0
PROOP

2 years ago

1291fcb5-2ee9-496e-8868-28df0eb6ae6a



2 years ago

Railway does not block anything of this sort unless there was an attack originating from Hetzner

what errors are you getting?


blankparticle
PRO

2 years ago

connection timeouts


mcpizza0
PROOP

2 years ago

deployments on railway cant access items in hetzner (services)
and hetzner services cant access a DB in railway


2 years ago

let keep this chat in here please, chit chat would not be the place


blankparticle
PRO

2 years ago

I just pointed it as it had some extra info


blankparticle
PRO

2 years ago

Will chat here


mcpizza0
PROOP

2 years ago

@Brody
connections time out on both sides:

1229814637843710000


mcpizza0
PROOP

2 years ago

happy to give the server IP if you could check internally for blocks


2 years ago

please note I don't work for Railway.

can you connect to the database locally?


mcpizza0
PROOP

2 years ago

ah sorry
saw a fancy color and assumed


mcpizza0
PROOP

2 years ago

yup


mcpizza0
PROOP

2 years ago

i can connect to the db from my local machine


mcpizza0
PROOP

2 years ago

connect to both sets of servers without issue


mcpizza0
PROOP

2 years ago

but they just wont communicate


2 years ago

are you sure you are using all the correct credentials?


blankparticle
PRO

2 years ago

we haven't changed it, it worked fine until yesterday


blankparticle
PRO

2 years ago

we also reverted all commits, so nothing in code too


2 years ago

can you ssh into the VPS and do some debugging, pinging, etc


mcpizza0
PROOP

2 years ago

yup


mcpizza0
PROOP

2 years ago

can even connect to it from other VPSs elsewhere


mcpizza0
PROOP

2 years ago

it feel like a firewall block


2 years ago

let's not jump to such conclusions just yet


2 years ago

are you able to ping that host from your vps (the same VPS that is having connection issues)


mcpizza0
PROOP

2 years ago

yes, the vps can ping the ip of the railway service its trying to connect to


2 years ago

can you use the mysql cli from within the VPS to connect to the database


mcpizza0
PROOP

2 years ago

lemme install and test


mcpizza0
PROOP

2 years ago

connection times out


mcpizza0
PROOP

2 years ago

can connect with the exact same parameters from my local machine


2 years ago

how have you been able to confirm that this issue is not with hetzner's network?


mcpizza0
PROOP

2 years ago

the vps can connect to another db hosted elsewhere without any issues


2 years ago

unfortunately that doesn't confirm that this isn't an issue with hetzner's network


mcpizza0
PROOP

2 years ago

do you have other ideas to test?


blankparticle
PRO

2 years ago

try with another vps


2 years ago

does this VPS have a static IP?


mcpizza0
PROOP

2 years ago

yes


mcpizza0
PROOP

2 years ago

I tried connecting from that VPS to another db in another railway environment
same issue


2 years ago

are you able to renew that static IP?


mcpizza0
PROOP

2 years ago

possibly


mcpizza0
PROOP

2 years ago

but would rather leave it as a last resort since theres lots tied to it


2 years ago

do you have a separate VPS on hetzner that you can try mysql from?


mcpizza0
PROOP

2 years ago

ill spin one up, give me a sec


mcpizza0
PROOP

2 years ago

yes, it connects fine


mcpizza0
PROOP

2 years ago

its also in the same DC as the failing server
but is using a different IP range


mcpizza0
PROOP

2 years ago

so it seems that Hetzner is not blocking railway


mcpizza0
PROOP

2 years ago

did it happen more than once ?


mcpizza0
PROOP

2 years ago

if yes, we need to move off railway
we have critical infrastructure that we cant swap out like this in production


2 years ago

at this time we don't have definitive proof of who's blocking who


2 years ago

may I ask why not move the services on the VPS to railway?


blankparticle
PRO

2 years ago

the services on vps needs multiple tcp ports, which railways fails to provide


2 years ago

you would need more than 1 TCP port per app?


2 years ago

ah so really it's neither gcp or hetzner's fault


mcpizza0
PROOP

2 years ago

that IP is reported as location germany on all the lists


2 years ago

not the list gcp uses apparently


mcpizza0
PROOP

2 years ago

will test from another vps in the same IP range


mcpizza0
PROOP

2 years ago

works fine from another IP in the same range


mcpizza0
PROOP

2 years ago

49.13.X.X


2 years ago

I would go with bart's proposed solution


mcpizza0
PROOP

2 years ago

its a possibility


mcpizza0
PROOP

2 years ago

but i'd like to find a more permanent solution since we have an identical setup for production


mcpizza0
PROOP

2 years ago

thanks for the help diagnosing though 💪


2 years ago

if gcp is blocking hetzner, there really nothing railway can do about that, they are at gcp's whim here


2 years ago

railway isn't blocking your singular static IP, so we can rule railway out of the equation


2 years ago

unless you tried to open well over 10k concurrent requests? this isn't a block from the railway side of things


mcpizza0
PROOP

2 years ago

yup, seems like its something upstream


2 years ago

so does a singular app of yours need to open more than one TCP port?


blankparticle
PRO

2 years ago

there are 2 apps which needs us to run vps, one needs a tpc port with a custom subdomain on our own and another with access to like 4 ports


mcpizza0
PROOP

2 years ago

we also need fixed IPs


mcpizza0
PROOP

2 years ago

its a mail system


mcpizza0
PROOP

2 years ago

signup btw… could prob get brody@ https://unn.sh


2 years ago

may I ask if you both are pro on railway? you seen to be missing the badges if so


blankparticle
PRO

2 years ago

yes we have a pro team


blankparticle
PRO

2 years ago

it didn't give use badges for some reason


2 years ago

now you have them


mcpizza0
PROOP

2 years ago

🔥


2 years ago

Google blocks a littany of IP addresses due to Hertzner propensity to host...very sketchy shitDDOS: https://managingwp.io/live-blog/google-cloud-and-hetzner-connectivity-issues-gridpane-specifics/Violating Iranian Sanctions: https://blog.cloud66.com/hetzner-connectivity-issues-due-to-sanction-busting-activitiesMore: https://github.com/kubernetes/kops/issues/16466


Loading...