Railway <> Hetzner servers cant communicate

mcpizza0PRO

a year ago

We had a deployment running succesfully for the last week

today, the railway instances cant communicate with the servers in Hetzner datacenter, and the otherway around is true too

is there some way we can diagnose this?

feels like railway has blocked the whole ip range

1 Replies

mcpizza0PRO

a year ago

1291fcb5-2ee9-496e-8868-28df0eb6ae6a



a year ago

Railway does not block anything of this sort unless there was an attack originating from Hetzner

what errors are you getting?


blankparticlePRO

a year ago

connection timeouts


mcpizza0PRO

a year ago

deployments on railway cant access items in hetzner (services)
and hetzner services cant access a DB in railway


a year ago

let keep this chat in here please, chit chat would not be the place


blankparticlePRO

a year ago

I just pointed it as it had some extra info


blankparticlePRO

a year ago

Will chat here


mcpizza0PRO

a year ago

@Brody
connections time out on both sides:

1229814637843710000


mcpizza0PRO

a year ago

happy to give the server IP if you could check internally for blocks


a year ago

please note I don't work for Railway.

can you connect to the database locally?


mcpizza0PRO

a year ago

ah sorry
saw a fancy color and assumed


mcpizza0PRO

a year ago

yup


mcpizza0PRO

a year ago

i can connect to the db from my local machine


mcpizza0PRO

a year ago

connect to both sets of servers without issue


mcpizza0PRO

a year ago

but they just wont communicate


a year ago

are you sure you are using all the correct credentials?


blankparticlePRO

a year ago

we haven't changed it, it worked fine until yesterday


blankparticlePRO

a year ago

we also reverted all commits, so nothing in code too


a year ago

can you ssh into the VPS and do some debugging, pinging, etc


mcpizza0PRO

a year ago

yup


mcpizza0PRO

a year ago

can even connect to it from other VPSs elsewhere


mcpizza0PRO

a year ago

it feel like a firewall block


a year ago

let's not jump to such conclusions just yet


a year ago

are you able to ping that host from your vps (the same VPS that is having connection issues)


mcpizza0PRO

a year ago

yes, the vps can ping the ip of the railway service its trying to connect to


a year ago

can you use the mysql cli from within the VPS to connect to the database


mcpizza0PRO

a year ago

lemme install and test


mcpizza0PRO

a year ago

connection times out


mcpizza0PRO

a year ago

can connect with the exact same parameters from my local machine


a year ago

how have you been able to confirm that this issue is not with hetzner's network?


mcpizza0PRO

a year ago

the vps can connect to another db hosted elsewhere without any issues


a year ago

unfortunately that doesn't confirm that this isn't an issue with hetzner's network


mcpizza0PRO

a year ago

do you have other ideas to test?


blankparticlePRO

a year ago

try with another vps


a year ago

does this VPS have a static IP?


mcpizza0PRO

a year ago

yes


mcpizza0PRO

a year ago

I tried connecting from that VPS to another db in another railway environment
same issue


a year ago

are you able to renew that static IP?


mcpizza0PRO

a year ago

possibly


mcpizza0PRO

a year ago

but would rather leave it as a last resort since theres lots tied to it


a year ago

do you have a separate VPS on hetzner that you can try mysql from?


mcpizza0PRO

a year ago

ill spin one up, give me a sec


mcpizza0PRO

a year ago

yes, it connects fine


mcpizza0PRO

a year ago

its also in the same DC as the failing server
but is using a different IP range


mcpizza0PRO

a year ago

so it seems that Hetzner is not blocking railway


mcpizza0PRO

a year ago

did it happen more than once ?


mcpizza0PRO

a year ago

if yes, we need to move off railway
we have critical infrastructure that we cant swap out like this in production


a year ago

at this time we don't have definitive proof of who's blocking who


a year ago

may I ask why not move the services on the VPS to railway?


blankparticlePRO

a year ago

the services on vps needs multiple tcp ports, which railways fails to provide


a year ago

you would need more than 1 TCP port per app?


a year ago

ah so really it's neither gcp or hetzner's fault


mcpizza0PRO

a year ago

that IP is reported as location germany on all the lists


a year ago

not the list gcp uses apparently


mcpizza0PRO

a year ago

will test from another vps in the same IP range


mcpizza0PRO

a year ago

works fine from another IP in the same range


mcpizza0PRO

a year ago

49.13.X.X


a year ago

I would go with bart's proposed solution


mcpizza0PRO

a year ago

its a possibility


mcpizza0PRO

a year ago

but i'd like to find a more permanent solution since we have an identical setup for production


mcpizza0PRO

a year ago

thanks for the help diagnosing though 💪


a year ago

if gcp is blocking hetzner, there really nothing railway can do about that, they are at gcp's whim here


a year ago

railway isn't blocking your singular static IP, so we can rule railway out of the equation


a year ago

unless you tried to open well over 10k concurrent requests? this isn't a block from the railway side of things


mcpizza0PRO

a year ago

yup, seems like its something upstream


a year ago

so does a singular app of yours need to open more than one TCP port?


blankparticlePRO

a year ago

there are 2 apps which needs us to run vps, one needs a tpc port with a custom subdomain on our own and another with access to like 4 ports


mcpizza0PRO

a year ago

we also need fixed IPs


mcpizza0PRO

a year ago

its a mail system


mcpizza0PRO

a year ago

signup btw… could prob get brody@ https://unn.sh


a year ago

may I ask if you both are pro on railway? you seen to be missing the badges if so


blankparticlePRO

a year ago

yes we have a pro team


blankparticlePRO

a year ago

it didn't give use badges for some reason


a year ago

now you have them


mcpizza0PRO

a year ago

🔥


10 months ago

Google blocks a littany of IP addresses due to Hertzner propensity to host...very sketchy shitDDOS: https://managingwp.io/live-blog/google-cloud-and-hetzner-connectivity-issues-gridpane-specifics/Violating Iranian Sanctions: https://blog.cloud66.com/hetzner-connectivity-issues-due-to-sanction-busting-activitiesMore: https://github.com/kubernetes/kops/issues/16466