Private network connectivity broken: Envoy cannot reach internal services despite correct DNS resolution
falconleandro
FREEOP

a month ago

Issue

Envoy gateway is unable to connect to internal PostgREST and Gotrue Auth services via Railway's private network, despite correct variable resolution and DNS configuration.

Diagnostic Summary

All variables resolve correctly to .railway.internal hostnames, but TCP connections from Envoy to other internal services timeout consistently.

Symptoms

  • Envoy service: Online, receiving requests correctly

    • GET / returns 401 (Studio basic auth, working)
    • GET /rest/v1/ returns 403 (PostgREST RBAC, working)
    • GET /auth/v1/health returns 503 upstream timeout (BROKEN)
    • GET /rest/v1/comercios returns 503 upstream timeout (BROKEN)
  • PostgREST service: Online, 1/1 running

  • Gotrue Auth service: Online, 1/1 running

Root Cause Evidence

  1. Variables resolve correctly to .railway.internal:

    REST_HOST=postgrest.railway.internal

    AUTH_HOST=gotrue-auth.railway.internal

  2. Envoy configuration is correct with proper cluster addresses

  3. DNS resolver configured: nameserver fd12::10

  4. Problem is network connectivity, not configuration:

    • Envoy receives requests (401/403 responses work)
    • Envoy resolves hostnames correctly
    • TCP connections to other services timeout
    • Affects ALL upstream services
    • Error: "upstream connect error... reset reason: connection timeout"

Details

  • Project: giving-empathy
  • Environment: production
  • Region: sfo

Conclusion

This appears to be a private network connectivity issue at the Railway infrastructure level. All services are properly configured and hostnames resolve correctly, but TCP connections timeout.

Solved

1 Replies

Railway
BOT

a month ago

Apologies for the trouble. We had an incident affecting private networking in US West that has since been resolved, which likely caused the upstream connection timeouts your Envoy service was seeing. Please retry your requests and confirm whether connectivity to your internal services has recovered. You can see the full timeline here: status.railway.com/incident/Z5Y3WO06.


Status changed to Awaiting User Response Railway • about 1 month ago


Railway
BOT

a month ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway • about 1 month ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...