Cannot connect to private services redis: dial tcp []:6379: i/o timeout

nic0l3333
PRO

9 months ago

I've asked similar questions before, on https://help.railway.app/questions/cannot-connect-to-internal-services-err-a96fa0e8, yes indeed, the issue was caused by the missing sleep 5s, after setting the sleep 5 in Dockerfile, issue was resolved.

However, this time the issue is different, even though with https://github.com/brody192/rueidis-test (which we tested before that works perfectly), the error:

error connecting to redis: dial tcp [fd12:9662:5d5c::87:89d8:743d]:6379: i/o timeout

is not solvable. I tried to increase the happy eyeballs fallback delay to 5s, made sure that the container will always get a 5s sleep, the error persist.

Such issue was appeared once after we migrated our projects from Hobby Plan to Pro Plan, as well as after we scaled our Redis instance storage from 5GB to 20GB.

Can you help me?

View Deploy details

ⓘ Deployment information is only viewable by project members and Railway employees.

Awaiting User Response

9 Replies

9 months ago

Quick ask- what happens when you connect via the public network?


Status changed to Awaiting User Response railway[bot] 9 months ago


nic0l3333
PRO

9 months ago

Fully working when connected through public network.


Status changed to Awaiting Railway Response railway[bot] 9 months ago


nic0l3333
PRO

9 months ago

And, another thing is, since we switched to public network, the amount of bandwidth cost increased to 16576GB from 2827GB. However, the total amount of our websites serves only 6GB, was this a correct value of how it was calculated?


9 months ago

Hey Nicole, so this confirms my suspicion on the location of the bug. I am going to raise this to the Infra team.

As for the bandwidth, I am going to see if we can waive the period of cost that you are facing, but it doesn't matter the storage in your Redis but the traffic that you are doing to that service, it seems like you are doing a lot of reads over the network hence the cost.


Status changed to Awaiting User Response railway[bot] 9 months ago


nic0l3333
PRO

9 months ago

Thanks for the reply!

Hey Nicole, so this confirms my suspicion on the location of the bug. I am going to raise this to the Infra team.

About the IPv6 issue...

We are trying to use Supabase's direct connection today, which is IPv6 only, unfortunately, we failed to connect, the error message indicates the "network unreachable".

I searched the support forum, learned that the Railway is fully IPv6 compatible, was this the same issue you mentioned? Was this the same bug?

As for the bandwidth, I am going to see if we can waive the period of cost that you are facing, but it doesn't matter the storage in your Redis but the traffic that you are doing to that service, it seems like you are doing a lot of reads over the network hence the cost.

Thanks for the comments, we have identified the reading issue from our side, it was indeed caused by the insane amount of full-amount of data reading from Redis.

We have fixed this bug, the bandwidth reduced, and everything looks normal now except the IPv6 internal network.

Hopefully this will not interfere any other services on Railway's side, or customers you have.

And, thanks for the waive check, it would be awesome if we could get some.


Status changed to Awaiting Railway Response railway[bot] 9 months ago


nic0l3333
PRO

9 months ago

For Redis connectivity issue, this is quite interesting, the network was working on our staging environment, but not for the production environment.


9 months ago

There is a known bug where the private network is not attached again after a volume migration, I assume that's what Angelo was getting at.

A redeploy of the database has always fixed this.


Status changed to Awaiting User Response railway[bot] 9 months ago


nic0l3333
PRO

9 months ago

So the recommended way of fixing this would be recreating of the Redis instance, right? Is the volume required to be deleted? (we still left gigabytes of data inside it)


Status changed to Awaiting Railway Response railway[bot] 9 months ago


9 months ago

Simply redeploy and the problem is fixed, based on my previous message, may I ask how you came to the conclusion that you would need to recreate anything?


Status changed to Awaiting User Response railway[bot] 9 months ago