What region is this deployed in?

us east

All requests are failing righ tnow

No errors on the application or db

Never had this issue before, any clues? could it be railway issue?

Some requests are now going through

Ok they aren't failing anymore

So this is most likely not an app issue on my end or atleast I can't see anything leading me to believe that

This happened again like 3 hours ago, went to bed, just woke up to many messages, so would like some confirmation from someone if this is on my side or railway if possible

Any ideas? It’s on and off

Would adding replicas help against this?

Idk how accurate this is since it's AI, but just want to try to figure out what caused this

Hi this is happening again currently

Pretty urgent, so if anyone has any ideas 🙂

Happening for us too

Scaling to 2 replicas didn't work, databases seem fine, redis seems fine, no clue what's going on

499 errors?

Yeah

anyone from the team could confirm if it's an app issue?

pls

Checking DB logs:
2026-05-12 16:49:27.871 UTC [37676] LOG: unexpected EOF on client connection with an open transaction 2026-05-12 16:49:27.880 UTC [37671] LOG: could not receive data from client: Connection reset by peer 2026-05-12 16:49:27.890 UTC [37675] LOG: could not receive data from client: Connection reset by peer 2026-05-12 16:49:27.884 UTC [37671] LOG: unexpected EOF on client connection with an open transaction 2026-05-12 16:49:27.890 UTC [37675] LOG: unexpected EOF on client connection with an open transaction 2026-05-12 16:49:28.634 UTC [37604] LOG: could not receive data from client: Connection reset by peer 2026-05-12 16:49:28.635 UTC [37682] LOG: could not receive data from client: Connection reset by peer

This was after a postgres restart

Redeploy the db now is just stuck on deploying and innaccessible

I am not using postgres btw. Could be an internal networking issue, since I'm using Dragonfly

one of my postgres wont even deploy anymore 🙁

here's more logs:

File "/app/.venv/lib/python3.10/site-packages/flask_cors/extension.py", line 176, in wrapped_function return cors_after_request(app.make_response(f(*args, **kwargs))) File "/app/.venv/lib/python3.10/site-packages/flask/app.py", line 865, in full_dispatch_request rv = self.ensure_sync(before_func)() g.db_conn = get_connection() File "/app/app/db.py", line 29, in get_connection _pools[host] = pool.ThreadedConnectionPool(1, 20, dsn) File "/app/.venv/lib/python3.10/site-packages/psycopg2/pool.py", line 59, in __init__ File "/app/.venv/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect psycopg2.OperationalError: connection to server at "postgres-88e0f11e.railway.internal" (fd12:685a:3643:0:a000:2f:ca06:13c5), port 5432 failed: Connection timed out Is the server running on that host and accepting TCP/IP connections?

still happening to you?

Yes

Switching to external redis connection for the time being

gonna try that

Didn't work for me, switched redis and postgres to public

I meant that I started a redis on Upstash and pointed my app to the external url

ahh

that fixed it?

Yeah

Something is breaking with the internal networking

ok i disabled all redis and seems to work

Still not working for me

ffs

Please can someone from the team take a look

it's now been an hour of constant 499 errors

The server works for the first 10 seconds after a redeploy, after that it goes abck to 499 errors, we've had this API running for a year+ and never have enoucntered these issues, nor have we changed anything recently to cause this

Okay this has escalated to full downtime now, is it possible to get someone on this?

^ Anyone

There's an AWS us-east-1 outage, could be because of that

do u have a link to that?

strange that would affect us tho

https://health.aws.amazon.com/health/status

Or Claude is lying Claude was lying

I don’t see it

I dont see it

Anyone? pls

@Alex Op seems to be recovering, you?

Yeah

Can anyone from the team advise? if it was an app issue it would most likely not get resolved by itself no?

!t

If anyone from the team does have an idea on the cause would love to know now, just don’t want it happening overnight again