Redis not connecting on Railway Metal

geongeorge
PRO

a month ago

Redis private URL connection stopped working after migrating to metal
Same Redis instance worked fine with private URL on legacy
Public URL still works on metal
Private URL credentials haven't changed

We tried with both consumer service in metal and gcp. Both does not work.

What we tested:
Private URL works: legacy Redis ✅
Private URL fails: metal Redis ❌
Public URL works: metal Redis ✅

Solved$10 Bounty

9 Replies

geongeorge
PRO

a month ago

Hi can someone help?

brody
EMPLOYEE

a month ago

I have bountied this thread so that the community can help!

rahrang
HOBBY

a month ago

Is there a more specific error you can share? There's this doc which has some common problems & solutions.

In my NodeJS application using Bull MQ, I had to modify the environment variable with the ?family=0 query.

REDIS_BULLMQ_URL="${{MY_REDIS_SERVICE.REDIS_URL}}?family=0"

geongeorge
PRO

a month ago

I have ?family=0 already.
There is no error.

geongeorge
PRO

a month ago

@shaheem_mpm Can you post if there is any error you saw?

Anonymous
PRO

a month ago

Error:

[ioredis] Unhandled error event: Error: connect ECONNREFUSED fd12:614f:b0aa::90:87c8:d447:6379

    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1636:16)

[ioredis] Unhandled error event: Error: connect ECONNREFUSED fd12:614f:b0aa::90:87c8:d447:6379

    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1636:16)

[ioredis] Unhandled error event: Error: connect ECONNREFUSED fd12:614f:b0aa::90:87c8:d447:6379

    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1636:16)

[ioredis] Unhandled error event: Error: connect ECONNREFUSED fd12:614f:b0aa::90:87c8:d447:6379

    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1636:16)

[ioredis] Unhandled error event: Error: connect ETIMEDOUT

    at Socket. (/app/node_modules/.pnpm/ioredis@5.4.2/node_modules/ioredis/built/Redis.js:170:41)

    at Object.onceWrapper (node:events:632:28)

    at Socket.emit (node:events:518:28)

    at Socket._onTimeout (node:net:609:8)

    at listOnTimeout (node:internal/timers:588:17)

    at processTimers (node:internal/timers:523:7)

[ioredis] Unhandled error event: Error: connect ETIMEDOUT

    at Socket. (/app/node_modules/.pnpm/ioredis@5.4.2/node_modules/ioredis/built/Redis.js:170:41)

    at Object.onceWrapper (node:events:632:28)

    at Socket.emit (node:events:518:28)

    at Socket._onTimeout (node:net:609:8)

    at listOnTimeout (node:internal/timers:588:17)

    at processTimers (node:internal/timers:523:7)

smolpaw
HOBBYTop 5% Contributor

a month ago

Is switching back to gcp possible and does it start working again once you do ?
This is not a suggestion to switch back to gcp. Just a question.

mittalyashu
HOBBY

a month ago

Can you try setting your env to with ?family=6:

```
REDIS_WORKER_URL=${{Redis BullMQ Worker.REDIS_URL}}?family=6
```

For internal connection Railway uses IPv6 and explicitly telling the redis to resolve the host accordingly.

geongeorge
PRO

a month ago

switching back to gcp worked! which is why I was able to confirm this is a metal issue

geongeorge
PRO

a month ago

And metal works using public url but that will just use up a lot of egres

prodstalker
HOBBY

a month ago

I had a similar problem in the past with a NestJS project.. I used ?family=0 to solve and make my app listen IPv6 like this

await app.listen(process.env.PORT ?? 3000, '::');

geongeorge
PRO

a month ago

Looks like it migrated again. Can you guys resolve the issue before this is migrated

1382299239266128000

geongeorge
PRO

a month ago

I have 2 services: remix js and nest js
I an emitting an event using ioredis to redis which is consumed by nest js

In both these servers the redis does not connect when I'm on railway metal

brody
EMPLOYEE

a month ago

Hello,

There are no known issues with connecting to Redis or any other services over the private network on Metal.

geongeorge
PRO

a month ago

I'm telling you it was working fine and now it isn't when I turn to metal! This is so frustrating. What do you need me to do? I can send you a loom video if you want.

Can you point me to a direction where I can check this?

brody
EMPLOYEE

a month ago

Redis in the staging environment, correct?

What service(s) are trying to connect to it, and where can I see the errors that are being omitted.

geongeorge
PRO

a month ago

Redis in all environment. This broke in production 2 times already because of automigation.

You are free to turn on and off on staging enviromnent if you want to check this.
Remix and Nest service is trying to connect to redis

brody
EMPLOYEE

a month ago

Where can I see the errors that are being logged? I am not seeing any Redis related errors from the Remix or Nest service in the staging environment.

geongeorge
PRO

a month ago

Give me a moment. Let me show you

geongeorge
PRO

a month ago

I'm not able to repro this now on staging. But we got it today on prod.

Staging: I have logs from previous deployments. WIll attach screenshot
Prod: Will show you the screenshots from the error after automigration today

geongeorge
PRO

a month ago

1382397024200032500

geongeorge
PRO

a month ago

Production. See this deployment: ce05dcc

1382397249497202700

brody
EMPLOYEE

a month ago

so to recap, at this moment everything is working?

geongeorge
PRO

a month ago

I'm testing on production now. It failed today morning as well

brody
EMPLOYEE

a month ago

what is your redis timeout set to?

geongeorge
PRO

a month ago

Confirmed I cannot repro this.

I have not set any timout. its the default timeout from ioredis which I believe is 1000ms

brody
EMPLOYEE

a month ago

could you try increasing that to 2s and report back if there are any further issues

geongeorge
PRO

a month ago

Sure. I will try. Thank you so much

minhlu
FREE

a month ago

hii, any updated with this?

testuser123
PROTop 10% Contributor

a month ago

Same problem here. Redis automigration to Metal causes 2 services connected to it to crash.

testuser123

Same problem here. Redis automigration to Metal causes 2 services connected to it to crash.

testuser123
PROTop 10% Contributor

a month ago

After 2h I finally fixed it.

testuser123

After 2h I finally fixed it.

charleslukes
PRO

25 days ago

whats your fix?

rendercoder
PRO

24 days ago

I encountered a similar problem, and it kept prompting a disk IO error. I tried to redeploy and it recovered.

brody
EMPLOYEE

22 days ago

!s

Status changed to Solved brody • 22 days ago

Redis not connecting on Railway Metal - Railway Help Station