Mixpost template outage
brokerkitservices
PROOP

7 months ago

# Railway Support: Redis IPv6 Connectivity Issue

**Project**: mixpost (`0fdaabcf-5cc8-473f-b8c5-fd27ce7bef9d`)
**Environment**: production
**Severity**: Critical - App Down 5+ hours
**URL**: https://social.brokerkit.app

## Problem

Mixpost container cannot connect to KeyDB (Redis) over private network. MySQL works fine, only Redis fails.

## Symptoms

```bash
# Application timeouts
curl -I https://social.brokerkit.app/mixpost/login
# Result: timeout after 10 seconds

# Redis connection error in logs
PhpRedisConnector.php line 181: Connection timed out

# DNS resolution fails
nc -zv keydb.railway.internal 6379
# Result: getaddrinfo: nodename nor servname provided, or not known

# Horizon crashes every 60 seconds
INFO exited: horizon_00 (exit status 1; not expected)

What Works ✅

  • MySQL: mysql.railway.internal:3306 connects instantly
  • KeyDB via TCP proxy: caboose.proxy.rlwy.net:20086 works
  • Container startup: All processes running (nginx, PHP-FPM)

What Fails ❌

  • Redis private network: keydb.railway.internal:6379 timeouts
  • DNS resolution: Only for KeyDB, not MySQL
  • All HTTP requests: Hang waiting for Redis

Configuration (Verified Correct)

KeyDB Service:

RAILWAY_PRIVATE_DOMAIN=keydb.railway.internal
KEYDB_URL=redis://default:<pwd>@keydb.railway.internal:6379

Mixpost Service:

REDIS_HOST=keydb.railway.internal
REDIS_PORT=6379
DB_HOST=mysql.railway.internal  # This works

Key Question

Why does mysql.railway.internal resolve/connect successfully but keydb.railway.internal fails DNS resolution? Both services are in same project/environment.

IPv6 Consideration

Railway docs say private networking is IPv6-only. PHP's Redis extension (phpredis) may not support IPv6 properly, but MySQL client does. Is this a known issue?

Diagnostics Run

  • Verified all environment variables correct
  • Confirmed KeyDB service is running and healthy
  • Tested MySQL (works) vs Redis (fails) from same container
  • KeyDB accessible via public TCP proxy (proves it's working)
  • Issue isolated to private network IPv6 connectivity

Timeline

  • Oct 19 19:16 UTC: Last successful deployment
  • Oct 20 ~20:00 UTC: Issue began (during AWS outage)
  • Oct 21 00:54 UTC: Root cause identified

Temporary Workaround

Using TCP proxy (not ideal):

REDIS_HOST=caboose.proxy.rlwy.net
REDIS_PORT=20086

This works but uses public internet and incurs egress costs.

Questions

  1. Why does KeyDB DNS fail when MySQL DNS works?
  2. Is there an IPv6 routing issue specific to Redis/KeyDB?
  3. Should we use ${{KeyDB.RAILWAY_PRIVATE_DOMAIN}} instead?
  4. Any recent private networking changes?

Supporting Info

Services:

  • Mixpost: 4ebcc492-214c-40fd-a122-6ddbea06d74e
  • KeyDB: 9892ea7a-413c-409c-81a0-e2d99d2dd09b
  • MySQL: 1ea08289-be8f-4a7b-9784-6c95d67858a3

Need guidance on fixing private network DNS resolution for Redis. Production app completely down.

$10 Bounty

2 Replies

Railway
BOT

7 months ago

Hey there! We've found the following might help you get unblocked faster:

If you find the answer from one of these, please let us know by solving the thread!


7 months ago

This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.

Status changed to Open brody 7 months ago


7 months ago

https://docs.railway.com/reference/errors/enotfound-redis-railway-internal#why-this-error-can-occur

Have you tried adding ?family=0 to the end of your Redis URI to force it to do an IPv6 DNS lookup?


Welcome!

Sign in to your Railway account to join the conversation.

Loading...