2 months ago
Environment
- Project ID: dead342b-9172-4d31-84a3-4b9e314121d9
- Region: asia-southeast1-eqsg3a
- Services Affected: ModdWell-QMS (PHP app), MySQL, Redis
Symptoms
1. MySQL Internal Connection Failure
SQLSTATE[HY000] [2002] Operation timed out
Host: mysql.railway.internal:3306
- All 30 retry attempts failed
- Connection never established
- Same issue persisted across 6+ deployment attempts
2. Redis Internal Connection Failure
RedisException: Operation timed out
Host: redis.railway.internal:6379
- Same timeout behavior as MySQL
3. Public Proxy URLs Work Fine
# MySQL public - WORKS
nc -zv shortline.proxy.rlwy.net 17211
# Connection succeeded!
# Redis public - WORKS
nc -zv mainline.proxy.rlwy.net 23552
# Connection succeeded!
Timeline (2026-01-03)
| Time (UTC) | Event |
|------------|-----------------------------------------------|
| 09:46 | First deployment failed - MySQL timeout |
| 09:54 | Redeploy failed - same issue |
| 10:37 | Redeploy failed - same issue |
| 11:05 | Redeploy failed - same issue |
| 16:17 | Redeploy failed - same issue |
| 16:24 | Redeploy failed - same issue |
| 16:27 | Switched to public URLs - MySQL connected |
| 16:29 | Redis still failing on internal URL |
| 16:30 | Switched Redis to public URL - both connected |
| 16:31 | Deployment healthy |
Workaround Applied
Changed environment variables to use public proxy URLs:
# MySQL (was: mysql.railway.internal:3306)
DB_HOST=shortline.proxy.rlwy.net
DB_PORT=17211
# Redis (was: redis.railway.internal:6379)
REDIS_HOST=mainline.proxy.rlwy.net
REDIS_PORT=23552
REDIS_URL=redis://default:***@mainline.proxy.rlwy.net:23552
Impact
- Multiple hours of deployment failures
- Added latency due to public proxy routing instead of internal networking
- Potential security concern (traffic now routes through public internet)
Request
Please investigate why internal DNS resolution or routing for *.railway.internal hostnames is failing in the asia-southeast1-eqsg3a region.
1 Replies
2 months ago
this is likely an IPv6 issue with PHP's mysql/redis extensions. railway's internal networking uses IPv6 by default, and PHP's PDO mysql driver has known inconsistencies with IPv6 address handling (some versions want brackets around the address, some don't). the fact that public URLs work confirms your app is fine and it's specifically the internal DNS/IPv6 resolution failing.
something to try when you want to switch back to internal networking: first, check if your project was created before october 2025 (legacy environments are IPv6 only, newer ones support both). add a 3 second sleep before your app starts connecting to databases because railway's internal DNS resolver isn't available for the first few seconds after container start.
for PHP specifically, make sure pdo_mysql and redis extensions are compiled with IPv6 support and try wrapping the IPv6 address in brackets if you're using the raw address. but honestly if public URLs are working and the latency is acceptable for your use case, that's a valid permanent solution too. railway is aware of intermittent private networking issues and is working on fixes.
so make sure u see :: not 0.0.0.0