Production service lost outbound connectivity after redeploy
story-boards-ai
PROOP

2 months ago

Our production Railway service ciaro-backend appears unable to make outbound connections.

From inside the running production container:

  • TCP connection to external Postgres times out: host: 13.57.136.47 port: 5432 error: connect ETIMEDOUT 13.57.136.47:5432
  • HTTPS fetch to Stytch fails: URL: https://api.stytch.com error: TypeError: fetch failed Redeploy/restart did not resolve it. After redeploy, startup still fails with: [Nest] ERROR [TypeOrmModule] Unable to connect to the database. Retrying (1)... Error: connect ETIMEDOUT 13.57.136.47:5432 Railway internal networking still works: the same service connects successfully to Redis via: redis://redis-tleo-prod.railway.internal:6379 This looks like outbound/external networking from the production runtime is broken, or the service is being routed through an egress path that external services reject/block. Project: Ciaro Pro Environment: production Service: ciaro-backend Can you check outbound egress/networking for this production service/runtime?
Solved

4 Replies

2 months ago

most likely related to:


story-boards-ai
PROOP

2 months ago

okay. switched that off. also switched regions. it recovered.

thats the second time we had issues in US West. Staying in US East for now


2 months ago

I don't think it's related to us-west, more so just static ips in general


We're aware of intermittent outbound connectivity drops affecting some services, where outbound connections fail while internal networking and DNS keep working, and we're actively investigating. If you hit it, the quickest workaround is to redeploy, which moves your service onto a fresh host and restores connectivity. If it happens again, reply here with the rough time it started and we'll jump on it while it's live. Those real-time reports are exactly what help us track down the cause.


Status changed to Awaiting User Response angelo-railway 24 days ago


Railway
BOT

17 days ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway 17 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...