Redis Socket Timeout
magickml
PROOP

2 years ago

Hey guys!
Been loving railway. Finalizing our production deployment for the release of our product. We have heavy reliance on Redis, and i have been noticing that our Redis instance seems to regularly kick off connected clients (ie our other services). Is there something I can set up for add to our redis instance to fix this?

Thanks!

0 Replies

magickml
PROOP

2 years ago

6fb8606f-116c-4e15-b653-24c48bdad9dd


magickml
PROOP

2 years ago

Any possible help on this? I tried migrating from Redis to IORedis and thought this was gone, but am still seeing it come up.


macwilko
HOBBY

2 years ago

I have a Read/Write timeout set to 3 seconds, that seems to work fine for my apps. I also connect to redis over private url.


magickml
PROOP

2 years ago

How did you configure your timeout?


magickml
PROOP

2 years ago

Any idea on this? The same code has no problem running against Redis cloud on heroku and other services. But for some reason my connections get kicked every 24-48 hours. And it happens at the same time in both my dev and prod environments.


brody
EMPLOYEE

2 years ago

the timeout would be configured through the redis client you are using


magickml
PROOP

2 years ago

<@539512869780455445> The error is actually not a socket timeout look at the logs. It is a 'connecton reset".


brody
EMPLOYEE

2 years ago

please have the redis client do connection retries


brody
EMPLOYEE

2 years ago

this is likely due to your app trying to write/read from a closed connection, either add retries or make sure to fully close idle connections


magickml
PROOP

2 years ago

I have a progressive backoff strategy implemented which tries to reconnect at every increasing periods of time. It looks like it manages to reconnect, and then gets booted again.


brody
EMPLOYEE

2 years ago

make sure to fully close idle connections


brody
EMPLOYEE

2 years ago

or you could try doing a ping every few minutes to keep the connection alive


magickml
PROOP

2 years ago

Every agent (AI Agent) we have in our system pings its health every few seconds which is picked up by an agent manage to ensure all enabled agents are healthy, so it is a very active redis instance, in theory.


magickml
PROOP

2 years ago

Looking through this thread right now:
https://github.com/redis/node-redis/issues/2032


brody
EMPLOYEE

2 years ago

are you using v4?


brody
EMPLOYEE

2 years ago

either way, these comments are for issues with redis database hosted in various places, meaning this isnt an isolated to railway issue


magickml
PROOP

2 years ago


magickml
PROOP

2 years ago

It seems to often be related to issues in the setup of the redis instance itself, at least in some cases. Just happened again.


magickml
PROOP

2 years ago

What is weird is that it happens at the exact time time in two separate environments on two different redis instances


magickml
PROOP

2 years ago

I havent had this issue on other managed redis services, so I may just swap out to use something like Upstash for now until I can determine what is going on. At this point there is not much left I can modify in my code to prevent it or handle it.


brody
EMPLOYEE

2 years ago

the github issues page you linked has some fixes


brody
EMPLOYEE

2 years ago

I'd also like to note that the redis on railway is not managed


magickml
PROOP

2 years ago

Thanks!


Loading...