Hey @prmeh! To be clear, when you enable static IP on your app in Railway, when that app makes a call out to an external resource, the request ends up timing out? Do all requests timeout, or is it intermittent? Are you able to view logs in the external resource, to confirm whether the requests are making it there or not?

Hi Melissa ,
We have multiple external services where we make calls to.
We talked to few of them and all calls made to those services are getting time out and the requests are not reaching them at all.
We were in call with those external services and they confirmed that they did not receive any requests from our end during a live on call testing we did with both parties.

Okay got it, thanks for the detail. And just to confirm, those external service providers aren't blocking the IP further upstream from the service? like at a firewall?

If we can just confirm that, I'll create a ticket for our infra team to look into it. It may also be helpful if you could provide an endpoint that we could test the connection with, is there an unauthenticated endpoint available?

Yes they dont have any blocking IP or firewall setup on their end.
Unfortunately there is no unauthenticated endpoint available.

So you will have to let me know once you have made some fix for us.

Just to be clear , we also have the same issue in our production environment as well. So once we fix it in this our acceptance environment then we need the same fix for production as well.

Thanks for the info, I'll open a ticket with infra to investigate.

Hi melissa, any updates on the ticket or investigation ? Its been 5 days

Hi there, our teams do triage today, so it will be discussed soon.
I will update you with any info I receive.

I am having issues with Static IPs as well. Intermittent timeout issues, we are trying to connect to the Public URL of a typesense service hosted in the same project as the service with the Static IP enabled. (Have to use the public URL for typesense because it doesn't listen on IPv6).

Failing:
Service w. Static IP > Typesense Public URL

As a workaround we are proxying our typesense queries through another service within the same Railway Project.

Workaround:

Service w. Static IP > Proxy Service w. NO Static IP > Typesense Public URL

Ah very weird, thanks for adding your note here @dane-stevens. My suspicion is these will have different root causes, but we can discuss them together.

I should note: we have enabled a static IP on that service to connect to a remote MSSQL database and whitelist the IP address. The connection to MSSQL directly to the IP address is stable, it's when we connect to any publicly accessible domain name hosted outside of Railway, or hosted on Railway within the same Project that the connection is unstable. My guess is that the issue is DNS related.

Very appreciative of the additional observations @dane. I have updated the ticket. Will let you both know what our infra team says in the next couple of hours.

@prmeh - we checked your static-ip enabled service, and it seems that the external resource is not allowing the request through. It appears those external services are deployed to AWS, we are not aware of any blocking of our IP on AWS, but you'll need to reach out to them to troubleshoot as we have no visibility. Otherwise, the container itself is having no issue connecting to outside resources.

@dane-stevens - can you please provide a link to the affected resource? if you prefer not to share in a public thread, you can open a private thread.

Hi Melissa,
It would be difficult to go to every external service we call and ask them if they use AWS or not ask them to whitelist us.
But I will try to get it in touch with one of them and see if they use AWS or any other cloud service provider and do whitelisting us provides any solution or not.

Few more questions
1. the static IP's created via railway .. are they IPv6 or IPv4 ?
2. Is there a way to get into the container and run commands for example traceroute integrations.eu.4shipping.com to see the call makes through ?

I don't think it's simply a matter of whitelisting. Services hosted on railway don't respond either, so unless railway is blocking railways own static IP's...

ok , so can you pls let us know what could be done here.

I'm not sure there is a whole lot that can be done. If there is a static IP available for the endpoints you are hitting, you should be able to use that instead of the url endpoints. I think railway needs to fix a DNS resolver issue though.

I'm seriously considering going back to hosting apps on regular old VMs. There are a bunch of open-source PaaS softwares now that take all the headache out of self hosting docker swarm.

Hi @prmeh, I understand you have many services affected, however, I would suggest starting with one to confirm if there is an issue on AWS. We did test from your container to confirm there were no issues with the static IP address connecting to other resources, only the ones you provided.

@dane-stevens - I had asked previously - can you please provide a link to the affected resource in Railway (the service which is timing out when attempting to call back into Railway from a static IP)? if you prefer not to share in a public thread, you can open a private thread.

@prmeh As a troubleshooting measure, we've updated your service to use the new runtime. Could you please redeploy at your convenience to see if that has an impact?

Hi Melissa,
Yes we restarted the services on our end and it seems to work on our all environments like acceptance and production as well.

So can you pls explain in detail what was changed exactly ?

Hello Pratik,

We found that your deployments where running on our Legacy runtime, the Legacy runtime does not play nicely with Static IPs.

We have blanket applied your services to the V2 runtime, and redeploying finalizes that switch over.

The blanket change applies to all services within production, staging, and PR envs created this month.

Dane, we have found the same issue applies to your services, and we have applied the same fix to them, both in your personal and team workspaces.

Thank you both for the report, and we are sorry about the inconvenience caused by this.

Thank you. It does seem to be working now!

Awsome!