Mixpost template outage
brokerkitservices
PROOP

2 months ago

# Railway Support: Redis IPv6 Connectivity Issue

**Project**: mixpost (`0fdaabcf-5cc8-473f-b8c5-fd27ce7bef9d`)
**Environment**: production
**Severity**: Critical - App Down 5+ hours
**URL**: https://social.brokerkit.app

## Problem

Mixpost container cannot connect to KeyDB (Redis) over private network. MySQL works fine, only Redis fails.

## Symptoms

```bash
# Application timeouts
curl -I https://social.brokerkit.app/mixpost/login
# Result: timeout after 10 seconds

# Redis connection error in logs
PhpRedisConnector.php line 181: Connection timed out

# DNS resolution fails
nc -zv keydb.railway.internal 6379
# Result: getaddrinfo: nodename nor servname provided, or not known

# Horizon crashes every 60 seconds
INFO exited: horizon_00 (exit status 1; not expected)
```

## What Works ✅

- MySQL: `mysql.railway.internal:3306` connects instantly
- KeyDB via TCP proxy: `caboose.proxy.rlwy.net:20086` works
- Container startup: All processes running (nginx, PHP-FPM)

## What Fails ❌

- Redis private network: `keydb.railway.internal:6379` timeouts
- DNS resolution: Only for KeyDB, not MySQL
- All HTTP requests: Hang waiting for Redis

## Configuration (Verified Correct)

**KeyDB Service**:
```
RAILWAY_PRIVATE_DOMAIN=keydb.railway.internal
KEYDB_URL=redis://default:<pwd>@keydb.railway.internal:6379
```

**Mixpost Service**:
```
REDIS_HOST=keydb.railway.internal
REDIS_PORT=6379
DB_HOST=mysql.railway.internal  # This works
```

## Key Question

Why does `mysql.railway.internal` resolve/connect successfully but `keydb.railway.internal` fails DNS resolution? Both services are in same project/environment.

## IPv6 Consideration

Railway docs say private networking is IPv6-only. PHP's Redis extension (phpredis) may not support IPv6 properly, but MySQL client does. Is this a known issue?

## Diagnostics Run

- Verified all environment variables correct
- Confirmed KeyDB service is running and healthy
- Tested MySQL (works) vs Redis (fails) from same container
- KeyDB accessible via public TCP proxy (proves it's working)
- Issue isolated to private network IPv6 connectivity

## Timeline

- Oct 19 19:16 UTC: Last successful deployment
- Oct 20 ~20:00 UTC: Issue began (during AWS outage)
- Oct 21 00:54 UTC: Root cause identified

## Temporary Workaround

Using TCP proxy (not ideal):
```bash
REDIS_HOST=caboose.proxy.rlwy.net
REDIS_PORT=20086
```

This works but uses public internet and incurs egress costs.

## Questions

1. Why does KeyDB DNS fail when MySQL DNS works?
2. Is there an IPv6 routing issue specific to Redis/KeyDB?
3. Should we use `${{KeyDB.RAILWAY_PRIVATE_DOMAIN}}` instead?
4. Any recent private networking changes?

## Supporting Info

**Services**:
- Mixpost: `4ebcc492-214c-40fd-a122-6ddbea06d74e`
- KeyDB: `9892ea7a-413c-409c-81a0-e2d99d2dd09b`
- MySQL: `1ea08289-be8f-4a7b-9784-6c95d67858a3`

Need guidance on fixing private network DNS resolution for Redis. Production app completely down.
$10 Bounty

2 Replies

Railway
BOT

2 months ago

Hey there! We've found the following might help you get unblocked faster:

If you find the answer from one of these, please let us know by solving the thread!


brody
EMPLOYEE

2 months ago

This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.

Status changed to Open brody about 2 months ago


https://docs.railway.com/reference/errors/enotfound-redis-railway-internal#why-this-error-can-occur

Have you tried adding ?family=0 to the end of your Redis URI to force it to do an IPv6 DNS lookup?


Loading...