network connectivity issue between services
therealtradex
PROOP

25 days ago

Project: velocity-funds-infra (651dbc6b-7a2b-4956-b251-87cec9230a51)

Environment: production (20beb7e7-08e4-4c9d-8ebe-a8a7b16f8585)

Issue: Worker service cannot connect to Redis service on internal network

Error: Error: connect ETIMEDOUT from ioredis

Both services: Same region (us-east4-eqdc4a), both online and healthy

Tested: Multiple URL formats, with/without authentication, hardcoded internal domain, reference variables

Result: All connection attempts timeout regardless of configuration

Solved$20 Bounty

Pinned Solution

Since every Redis URL variation is timing out, I would stop focusing on connection strings and verify the network path itself.

Please work through these checks:

  1. Verify Redis is healthy

    • Check Redis logs and confirm it has started successfully and is accepting connections.
    • If Redis is restarting or crashing, fix that first.
  2. Verify internal networking

    • Open both the Worker and Redis services.
    • Confirm they are in the same environment and region.
    • Ensure internal/private networking is enabled.
  3. Test DNS resolution from the Worker

    
    nslookup redis
    

    or

    
    getent hosts redis
    

    If the hostname cannot be resolved, the issue is networking or service discovery.

  4. Test raw TCP connectivity

    
    nc -vz redis 6379
    

    If this times out, the Worker cannot reach Redis over the internal network.

  5. Verify environment variables

    
    echo $REDIS_HOST
    
    echo $REDIS_PORT
    

    Make sure the values are resolved correctly and are not still showing template/reference syntax.

  6. Test without ioredis

    Create a minimal TCP connection test using Node's built-in net module. If a plain TCP connection also times out, the problem is below the Redis client layer.

Key point:

  • Authentication issues usually return NOAUTH or WRONGPASS.
  • DNS issues usually return ENOTFOUND.
  • ETIMEDOUT generally means the connection never reached Redis at all.

If nc -vz redis 6379 also times out, this strongly points to an internal networking issue rather than a Redis configuration issue.

4 Replies

Railway
BOT

25 days ago

This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.

Status changed to Open Railway 25 days ago


Since every Redis URL variation is timing out, I would stop focusing on connection strings and verify the network path itself.

Please work through these checks:

  1. Verify Redis is healthy

    • Check Redis logs and confirm it has started successfully and is accepting connections.
    • If Redis is restarting or crashing, fix that first.
  2. Verify internal networking

    • Open both the Worker and Redis services.
    • Confirm they are in the same environment and region.
    • Ensure internal/private networking is enabled.
  3. Test DNS resolution from the Worker

    
    nslookup redis
    

    or

    
    getent hosts redis
    

    If the hostname cannot be resolved, the issue is networking or service discovery.

  4. Test raw TCP connectivity

    
    nc -vz redis 6379
    

    If this times out, the Worker cannot reach Redis over the internal network.

  5. Verify environment variables

    
    echo $REDIS_HOST
    
    echo $REDIS_PORT
    

    Make sure the values are resolved correctly and are not still showing template/reference syntax.

  6. Test without ioredis

    Create a minimal TCP connection test using Node's built-in net module. If a plain TCP connection also times out, the problem is below the Redis client layer.

Key point:

  • Authentication issues usually return NOAUTH or WRONGPASS.
  • DNS issues usually return ENOTFOUND.
  • ETIMEDOUT generally means the connection never reached Redis at all.

If nc -vz redis 6379 also times out, this strongly points to an internal networking issue rather than a Redis configuration issue.


I suggest you try to increase the connectionTimout for ioredis (set it to >=10000 for example). You can also try to use the public URL just to ensure the service is healthy and responding.


pancharasshubham

Since every Redis URL variation is timing out, I would stop focusing on connection strings and verify the network path itself. Please work through these checks: 1. Verify Redis is healthy * Check Redis logs and confirm it has started successfully and is accepting connections. * If Redis is restarting or crashing, fix that first. 2. Verify internal networking * Open both the Worker and Redis services. * Confirm they are in the same environment and region. * Ensure internal/private networking is enabled. 3. Test DNS resolution from the Worker ```bash nslookup redis ``` or ```bash getent hosts redis ``` If the hostname cannot be resolved, the issue is networking or service discovery. 4. Test raw TCP connectivity ```bash nc -vz redis 6379 ``` If this times out, the Worker cannot reach Redis over the internal network. 5. Verify environment variables ```bash echo $REDIS_HOST echo $REDIS_PORT ``` Make sure the values are resolved correctly and are not still showing template/reference syntax. 6. Test without ioredis Create a minimal TCP connection test using Node's built-in net module. If a plain TCP connection also times out, the problem is below the Redis client layer. Key point: * Authentication issues usually return NOAUTH or WRONGPASS. * DNS issues usually return ENOTFOUND. * ETIMEDOUT generally means the connection never reached Redis at all. If `nc -vz redis 6379` also times out, this strongly points to an internal networking issue rather than a Redis configuration issue.

therealtradex
PROOP

25 days ago

Resolved, thank you!


Thanks for the update.

@darseen Increasing connectionTimeout and testing via the public URL are good troubleshooting steps to help distinguish between a networking issue and a Redis service issue.

@therealtradex Glad you got it resolved, and thanks for sharing.


Status changed to Solved passos 24 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...