Difficulty connecting to redis

brent-hooverPRO

8 months ago

Trying to get my server running on RW and I am running into the issue with not being able to connect to redis no matter what I do. The error is just a timeout:

node:events:495

throw er; // Unhandled 'error' event

^

Error: connect ETIMEDOUT

at TLSSocket.<anonymous> (/var/lib/merchstack/node_modules/ioredis/built/Redis.js:170:41)

at Object.onceWrapper (node:events:631:28)

at TLSSocket.emit (node:events:517:28)

at TLSSocket.emit (node:domain:489:12)

at Socket._onTimeout (node:net:598:8)

at listOnTimeout (node:internal/timers:569:17)

at process.processTimers (node:internal/timers:512:7)

Emitted 'error' event on Keyv instance at:

at KeyvRedis.<anonymous> (/var/lib/merchstack/node_modules/keyv/src/index.js:60:46)

at KeyvRedis.emit (node:events:517:28)

at KeyvRedis.emit (node:domain:489:12)

at EventEmitter.<anonymous> (/var/lib/merchstack/node_modules/@keyv/redis/dist/index.js:21:48)

at EventEmitter.emit (node:events:517:28)

at EventEmitter.emit (node:domain:489:12)

at EventEmitter.silentEmit (/var/lib/merchstack/node_modules/ioredis/built/Redis.js:464:30)

at /var/lib/merchstack/node_modules/ioredis/built/redis/event_handler.js:196:14

at TLSSocket.<anonymous> (/var/lib/merchstack/node_modules/ioredis/built/Redis.js:177:61)

at Object.onceWrapper (node:events:631:28) {

errorno: 'ETIMEDOUT',

code: 'ETIMEDOUT',

syscall: 'connect'

}

Node.js v18.18.2

I have verified that the correct env vars are set and being used. I have also added family=0 to the configuration as I saw in another thread that you need that for ipv6. But still no luck connecting.

Any hints on where to look next would be appreciated.

Awaiting User Response

20 Replies

brent-hooverPRO

8 months ago

To aid in debugging and separating config issues from code issues I created a minimal NestJS/BullMQ app which is available here:

https://github.com/brent-hoover/minimal-nest-app

Using the REDIS_URL env var, this app also cannot connect and times out.


8 months ago

A minimal reproducible example helps tremendously, thank you! I'll try deploying this and work out what solution is needed and let you know.


Status changed to Awaiting User Response railway[bot] 8 months ago


8 months ago

I wasn't able to get any errors from that example, I've simply set REDIS_URL=${{Redis.REDIS_URL}}

With that said your linked service does not have a REDIS_URL variable.


brent-hooverPRO

8 months ago

It does in fact have that set:



When you say you "didn't get any errors", did you hit the endpoint that triggers BullMQ? That's when the error appears

Attachments


Status changed to Awaiting Railway Response railway[bot] 8 months ago


brent-hooverPRO

8 months ago

Here's the link to the minimal version I have deployed in my env. https://railway.app/project/8e51cf18-c9bd-48cf-a4fe-dac9783fbb1b/service/8ac93c68-7fb3-47dc-97fa-852aaa64703b. You can look at the deploy logs to see the error


8 months ago

Ah gotcha, I was looking at merchstack-server since that's what this thread was linked to, my bad.

I also did not hit any endpoints, I will try that now!


Status changed to Awaiting User Response railway[bot] 8 months ago


brent-hooverPRO

8 months ago

Yeah, my mistake for not pointing out how to trigger the error because I assume you could just look at the logs on my instance.


Status changed to Awaiting Railway Response railway[bot] 8 months ago


8 months ago

I can, just wasn't obvious what triggered it.


Status changed to Awaiting User Response railway[bot] 8 months ago


8 months ago

I was able to get your example to work with the changes made here - https://github.com/brody192/minimal-nest-app/commit/a084747c794a4d9c31dc2f4737199b7ef7545026

Attachments


brent-hooverPRO

8 months ago

Got it, will try that in the server app.


Status changed to Awaiting Railway Response railway[bot] 8 months ago


brent-hooverPRO

8 months ago

FYI that we currently run this software on k8's on AWS as a SaaS. A couple of enterprise customers have asked for the ability to self-host because they have internal rules about not using Amazon. So I trying to investigate if I can get it running here w/o having to make a lot of significant changes. Currently the app uses separate env vars for REDIS_HOST, REDIS_PORT, etc. Not sure why it was set up that way but it is.


brent-hooverPRO

8 months ago

Unfortunately no change to the actual app but that rules out any sort of networking etc.


8 months ago

Well the example app you gave me now works, so perhaps you have package version mismatch or the bull client configured incorrectly?


Status changed to Awaiting User Response railway[bot] 8 months ago


brent-hooverPRO

8 months ago

If I connect using the method you prescribe there, the app also crashes like this locally so it's definitely something in the app not liking the config that way


Status changed to Awaiting Railway Response railway[bot] 8 months ago


brent-hooverPRO

8 months ago

Now I am getting this error when using the REDIS_URL. I have updated the example app to use the exact same version of bullmq and everything works. But then the same parameters in our app give me this error. Obviously it's somehow in our app but any ideas what I should be looking for? It seems like it can't find redis.railway.internal and the fix for that before was family: 0 but as you can see from the logging at the top of the logs I am passing in that parameter.

EventEmitter.silentEmit (/var/lib/merchstack/node_modules/ioredis/built/Redis.js:464:30)

at Socket.<anonymous> (/var/lib/merchstack/node_modules/ioredis/built/redis/event_handler.js:196:14)

at Object.onceWrapper (node:events:632:26)

at Socket.emit (node:events:529:35) {

errno: -3007,

code: 'ENOTFOUND',

syscall: 'getaddrinfo',

hostname: 'redis.railway.internal'

}


8 months ago

Please note that you are using bull, not bullmq, they are different packages.

Are you getting this error locally or when on Railway?


Status changed to Awaiting User Response railway[bot] 8 months ago


brent-hooverPRO

8 months ago

No, I switched the example app to bullmq when I realized that I had the wrong package in the example app because bullmq is what we use in the real app. It didn't seem to make any difference however.

I am getting the error only on RW. I don't have any difficulty connecting locally or on our production deployment on AWS. I modified the app code to use REDIS_URL if it exists and parse it exactly like you did in the example app and still not working.

Note that there has been two different issues. The timeout issue seemed to be around auth as our current deployment does not use a USERNAME parameter so I was able to resolve that. But this cannot find issue seems to be persistent.

Here is the relevant snippet of code:

if (process.env.REDIS_URL) { console.log('Initializing RedisPlugin using REDIS_URL'); // init using REDIS_URL rather than components const redisURL = new URL(process.env.REDIS_URL); const connection = { host: redisURL.hostname, port: Number(redisURL.port), username: redisURL.username, password: redisURL.password, family: 0, };


Status changed to Awaiting Railway Response railway[bot] 8 months ago


8 months ago

Can your example app replicate the main issue you are seeing?

Since this is an application level issue we would need a fully reproducible example to proceed.


Status changed to Awaiting User Response railway[bot] 8 months ago


brent-hooverPRO

8 months ago

Yeah, this is an app issue not a Railway issue for sure. Is there any way to get a non ipv6 url to connect to (that's not publicly exposed) rather them me having to reverse engineer the entire app. I updated the example app to the exact same version of Nest and BullMQ and still no issue. I expect somewhere in the app it's stripping out the family: 0 parameter because it's just not expecting it.


Status changed to Awaiting Railway Response railway[bot] 8 months ago


8 months ago

Railway's private network is IPv6 only, we have no way of getting you a private domain on IPv4.


Status changed to Awaiting User Response railway[bot] 8 months ago


Difficulty connecting to redis - Railway Help Station