25 days ago
Project: velocity-funds-infra (651dbc6b-7a2b-4956-b251-87cec9230a51)
Environment: production (20beb7e7-08e4-4c9d-8ebe-a8a7b16f8585)
Issue: Worker service cannot connect to Redis service on internal network
Error: Error: connect ETIMEDOUT from ioredis
Both services: Same region (us-east4-eqdc4a), both online and healthy
Tested: Multiple URL formats, with/without authentication, hardcoded internal domain, reference variables
Result: All connection attempts timeout regardless of configuration
Pinned Solution
25 days ago
Since every Redis URL variation is timing out, I would stop focusing on connection strings and verify the network path itself.
Please work through these checks:
-
Verify Redis is healthy
- Check Redis logs and confirm it has started successfully and is accepting connections.
- If Redis is restarting or crashing, fix that first.
-
Verify internal networking
- Open both the Worker and Redis services.
- Confirm they are in the same environment and region.
- Ensure internal/private networking is enabled.
-
Test DNS resolution from the Worker
nslookup redisor
getent hosts redisIf the hostname cannot be resolved, the issue is networking or service discovery.
-
Test raw TCP connectivity
nc -vz redis 6379If this times out, the Worker cannot reach Redis over the internal network.
-
Verify environment variables
echo $REDIS_HOST echo $REDIS_PORTMake sure the values are resolved correctly and are not still showing template/reference syntax.
-
Test without ioredis
Create a minimal TCP connection test using Node's built-in net module. If a plain TCP connection also times out, the problem is below the Redis client layer.
Key point:
- Authentication issues usually return NOAUTH or WRONGPASS.
- DNS issues usually return ENOTFOUND.
- ETIMEDOUT generally means the connection never reached Redis at all.
If nc -vz redis 6379 also times out, this strongly points to an internal networking issue rather than a Redis configuration issue.
4 Replies
25 days ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • 25 days ago
25 days ago
Since every Redis URL variation is timing out, I would stop focusing on connection strings and verify the network path itself.
Please work through these checks:
-
Verify Redis is healthy
- Check Redis logs and confirm it has started successfully and is accepting connections.
- If Redis is restarting or crashing, fix that first.
-
Verify internal networking
- Open both the Worker and Redis services.
- Confirm they are in the same environment and region.
- Ensure internal/private networking is enabled.
-
Test DNS resolution from the Worker
nslookup redisor
getent hosts redisIf the hostname cannot be resolved, the issue is networking or service discovery.
-
Test raw TCP connectivity
nc -vz redis 6379If this times out, the Worker cannot reach Redis over the internal network.
-
Verify environment variables
echo $REDIS_HOST echo $REDIS_PORTMake sure the values are resolved correctly and are not still showing template/reference syntax.
-
Test without ioredis
Create a minimal TCP connection test using Node's built-in net module. If a plain TCP connection also times out, the problem is below the Redis client layer.
Key point:
- Authentication issues usually return NOAUTH or WRONGPASS.
- DNS issues usually return ENOTFOUND.
- ETIMEDOUT generally means the connection never reached Redis at all.
If nc -vz redis 6379 also times out, this strongly points to an internal networking issue rather than a Redis configuration issue.
25 days ago
I suggest you try to increase the connectionTimout for ioredis (set it to >=10000 for example). You can also try to use the public URL just to ensure the service is healthy and responding.
pancharasshubham
Since every Redis URL variation is timing out, I would stop focusing on connection strings and verify the network path itself. Please work through these checks: 1. Verify Redis is healthy * Check Redis logs and confirm it has started successfully and is accepting connections. * If Redis is restarting or crashing, fix that first. 2. Verify internal networking * Open both the Worker and Redis services. * Confirm they are in the same environment and region. * Ensure internal/private networking is enabled. 3. Test DNS resolution from the Worker ```bash nslookup redis ``` or ```bash getent hosts redis ``` If the hostname cannot be resolved, the issue is networking or service discovery. 4. Test raw TCP connectivity ```bash nc -vz redis 6379 ``` If this times out, the Worker cannot reach Redis over the internal network. 5. Verify environment variables ```bash echo $REDIS_HOST echo $REDIS_PORT ``` Make sure the values are resolved correctly and are not still showing template/reference syntax. 6. Test without ioredis Create a minimal TCP connection test using Node's built-in net module. If a plain TCP connection also times out, the problem is below the Redis client layer. Key point: * Authentication issues usually return NOAUTH or WRONGPASS. * DNS issues usually return ENOTFOUND. * ETIMEDOUT generally means the connection never reached Redis at all. If `nc -vz redis 6379` also times out, this strongly points to an internal networking issue rather than a Redis configuration issue.
25 days ago
Resolved, thank you!
25 days ago
Thanks for the update.
@darseen Increasing connectionTimeout and testing via the public URL are good troubleshooting steps to help distinguish between a networking issue and a Redis service issue.
@therealtradex Glad you got it resolved, and thanks for sharing.
Status changed to Solved passos • 24 days ago
