Redis not connecting on Railway Metal
geongeorge
PROOP

9 months ago

Redis private URL connection stopped working after migrating to metal
Same Redis instance worked fine with private URL on legacy
Public URL still works on metal
Private URL credentials haven't changed

We tried with both consumer service in metal and gcp. Both does not work.

What we tested:
Private URL works: legacy Redis ✅
Private URL fails: metal Redis ❌
Public URL works: metal Redis ✅

Solved$10 Bounty

35 Replies

geongeorge
PROOP

9 months ago

Hi can someone help?

brody
EMPLOYEE

9 months ago

I have bountied this thread so that the community can help!

rahrang
HOBBY

9 months ago

Is there a more specific error you can share? There's this doc which has some common problems & solutions.

In my NodeJS application using Bull MQ, I had to modify the environment variable with the ?family=0 query.

REDIS_BULLMQ_URL="${{MY_REDIS_SERVICE.REDIS_URL}}?family=0"

geongeorge
PROOP

9 months ago

I have ?family=0 already.
There is no error.

geongeorge
PROOP

9 months ago

@shaheem_mpm Can you post if there is any error you saw?

Anonymous
PRO

9 months ago

Error:

[ioredis] Unhandled error event: Error: connect ECONNREFUSED fd12:614f:b0aa::90:87c8:d447:6379

    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1636:16)

[ioredis] Unhandled error event: Error: connect ECONNREFUSED fd12:614f:b0aa::90:87c8:d447:6379

    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1636:16)

[ioredis] Unhandled error event: Error: connect ECONNREFUSED fd12:614f:b0aa::90:87c8:d447:6379

    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1636:16)

[ioredis] Unhandled error event: Error: connect ECONNREFUSED fd12:614f:b0aa::90:87c8:d447:6379

    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1636:16)

[ioredis] Unhandled error event: Error: connect ETIMEDOUT

    at Socket. (/app/node_modules/.pnpm/ioredis@5.4.2/node_modules/ioredis/built/Redis.js:170:41)

    at Object.onceWrapper (node:events:632:28)

    at Socket.emit (node:events:518:28)

    at Socket._onTimeout (node:net:609:8)

    at listOnTimeout (node:internal/timers:588:17)

    at processTimers (node:internal/timers:523:7)

[ioredis] Unhandled error event: Error: connect ETIMEDOUT

    at Socket. (/app/node_modules/.pnpm/ioredis@5.4.2/node_modules/ioredis/built/Redis.js:170:41)

    at Object.onceWrapper (node:events:632:28)

    at Socket.emit (node:events:518:28)

    at Socket._onTimeout (node:net:609:8)

    at listOnTimeout (node:internal/timers:588:17)

    at processTimers (node:internal/timers:523:7)
smolpaw
HOBBY

9 months ago

Is switching back to gcp possible and does it start working again once you do ?
This is not a suggestion to switch back to gcp. Just a question.

mittalyashu
PRO

9 months ago

Can you try setting your env to with ?family=6:

```
REDIS_WORKER_URL=${{Redis BullMQ Worker.REDIS_URL}}?family=6
```

For internal connection Railway uses IPv6 and explicitly telling the redis to resolve the host accordingly.

geongeorge
PROOP

9 months ago

switching back to gcp worked! which is why I was able to confirm this is a metal issue

geongeorge
PROOP

9 months ago

And metal works using public url but that will just use up a lot of egres

prodstalker
HOBBY

9 months ago

I had a similar problem in the past with a NestJS project.. I used ?family=0 to solve and make my app listen IPv6 like this

await app.listen(process.env.PORT ?? 3000, '::');

geongeorge
PROOP

9 months ago

Looks like it migrated again. Can you guys resolve the issue before this is migrated

1382299239266128000

geongeorge
PROOP

9 months ago

I have 2 services: remix js and nest js
I an emitting an event using ioredis to redis which is consumed by nest js

In both these servers the redis does not connect when I'm on railway metal

brody
EMPLOYEE

9 months ago

Hello,

There are no known issues with connecting to Redis or any other services over the private network on Metal.

geongeorge
PROOP

9 months ago

I'm telling you it was working fine and now it isn't when I turn to metal! This is so frustrating. What do you need me to do? I can send you a loom video if you want.

Can you point me to a direction where I can check this?

brody
EMPLOYEE

9 months ago

Redis in the staging environment, correct?

What service(s) are trying to connect to it, and where can I see the errors that are being omitted.

geongeorge
PROOP

9 months ago

Redis in all environment. This broke in production 2 times already because of automigation.

You are free to turn on and off on staging enviromnent if you want to check this.
Remix and Nest service is trying to connect to redis

brody
EMPLOYEE

9 months ago

Where can I see the errors that are being logged? I am not seeing any Redis related errors from the Remix or Nest service in the staging environment.

geongeorge
PROOP

9 months ago

Give me a moment. Let me show you

geongeorge
PROOP

9 months ago

I'm not able to repro this now on staging. But we got it today on prod.

Staging: I have logs from previous deployments. WIll attach screenshot
Prod: Will show you the screenshots from the error after automigration today

geongeorge
PROOP

9 months ago

1382397024200032500

geongeorge
PROOP

9 months ago

Production. See this deployment: ce05dcc

1382397249497202700

brody
EMPLOYEE

9 months ago

so to recap, at this moment everything is working?

geongeorge
PROOP

9 months ago

I'm testing on production now. It failed today morning as well

brody
EMPLOYEE

9 months ago

what is your redis timeout set to?

geongeorge
PROOP

9 months ago

Confirmed I cannot repro this.

I have not set any timout. its the default timeout from ioredis which I believe is 1000ms

brody
EMPLOYEE

9 months ago

could you try increasing that to 2s and report back if there are any further issues

geongeorge
PROOP

9 months ago

Sure. I will try. Thank you so much

minhlu
FREE

9 months ago

hii, any updated with this?

testuser123
PRO

9 months ago

Same problem here. Redis automigration to Metal causes 2 services connected to it to crash.

testuser123

Same problem here. Redis automigration to Metal causes 2 services connected to it to crash.

testuser123
PRO

9 months ago

After 2h I finally fixed it.

testuser123

After 2h I finally fixed it.

charleslukes
PRO

9 months ago

whats your fix?

rendercoder
PRO

9 months ago

I encountered a similar problem, and it kept prompting a disk IO error. I tried to redeploy and it recovered.

brody
EMPLOYEE

9 months ago

!s

Status changed to Solved brody • 9 months ago

Loading...