Are there any errors in the logs? If so, can you provide them?

yes here its saying it can "[DiagnosticLogger] Failed to flush logs: Error: Connection terminated due to connection timeout" this is just 1 of a million errors my entire company is crashed and down right now. i am compiling a rpeort to send you

Yes — Node pg errors on the app service, across every pool, from ~00:05 to ~00:24 UTC (Aug 29):

2026-08-29T00:20:13Z [err] [SiteAnalytics] Tracking error: Error: Connection terminated due to connection timeout

at /app/node_modules/pg-pool/index.js:45:11

[cause]: Error: Connection terminated unexpectedly

at Connection.<anonymous> (/app/node_modules/pg/lib/client.js:177:73)

2026-08-29T00:20:21Z [err] [Error] 500: timeout exceeded when trying to connect

[pg-pool exhausted: 155 more suppressed in last 10s]

pools{web:34t/0i/26w/34max dialer:8t/0i/1w/8max session:6t/0i/6w/6max bg:6t/0i/0w/12max int:0t/0i/0w/3max}

What they show: new connections to Postgres failed during establishment ("Connection terminated

due to connection timeout" wrapping "Connection terminated unexpectedly"), while already-checked-out

connections hung. All five of our pg pools went to 0 idle simultaneously and every request 500'd.

The app container was healthy the whole time — internal timers fired on schedule, no restart, CPU

and memory normal. It simply could not reach Postgres.

Timing: this began ~00:05 UTC, six minutes after incident 8GL2R2U5 ("Deployments slow to start",

opened 23:59 UTC, root-caused as "a networking issue on a single host"). We restarted the Postgres

service twice during the window with no effect; the platform recovered ~00:20–00:24 UTC, right as

your status page posted that the affected host was removed.

This is also the second DB-unreachable event on this project today. From 15:02:17 to 15:03:34.5 UTC,

~30 Postgres backends logged "canceling statement due to statement timeout" within the same 300 ms

window, even though their statements had started at very different times and were 30–50 s past the

30 s statement_timeout — the backends could not process cancel interrupts, which points to an

uninterruptible I/O wait at the host/volume level. A checkpoint write at 14:20 UTC took 68 s

(normally 0.1–2 s). Postgres never OOMed, never restarted, and was far under its CPU/memory limits

in both windows.

Project: 5255cfae-7e88-41a9-b9bc-399c79e30b1e (production — GameForce app + Postgres,

volume postgres-volume, 250GB provisioned / ~48GB used)

Three questions:

1. Was our Postgres service or its volume on (or network-adjacent to) the host removed in

   incident 8GL2R2U5?

2. Can you check for host/volume events (I/O stall, throttling, live migration, maintenance)

   affecting this Postgres at 15:02–15:05 UTC and 00:05–00:24 UTC today?

3. What are the IOPS/throughput ceilings on postgres-volume, and did we hit them in either window?

Errors in the Postgres logs*

Same here, my database is not connecting

Sounds like another railway crash?? Were you doing updates? i was not.

Please create your own thread.

What do we do? is this railway or us ?? Nothing is connecting

No, I was actually on my day off. Now I'm here trying to understand the problem. @0x5b62656e5d Ok, I will create the thread.

Are there any errors in your Postgres service?

yes my entire site and company is crashed! Nothing will connect. I have clients too! I have restarted it etc. I have done no updates or anyhting!

That doesn’t answer my question. Are there any errors present in the logs of your database service?

I restarted my Postgres, it never connects. If I go to "Console" tab it says "Your service's container is not running (status: exited). Deploy or restart your service, then try again."

yes datbase has x connection wont connect and

▎ Yes. During tonight's window (Aug 29 00:05–00:21 UTC) the Postgres service logs show repeated errors of two kinds:

▎

▎ ERROR: canceling statement due to lock timeout — e.g. 00:09:26.652 UTC [66389] CONTEXT: while locking tuple (61,3) in relation "users"; 00:09:31.805 [66368] and 00:09:41.299 [66403] on relation "users"; 00:09:33.299 [66346] on "agent_online_status"; 00:17:50.072 and 00:17:55.075 on "power_dialer_sessions".

▎

▎ ERROR: canceling statement due to statement timeout — bursts of these, e.g. five backends [66272/66313/66314/66349/66402] all cancelling within ~3ms of each other at 00:10:25.36 UTC.

▎

▎ These are our own configured timeouts firing (statement_timeout 30s, lock_timeout 4s) — the notable part is the pattern: plain reads and single-row writes that normally run in milliseconds were stalling past those bounds while the Postgres container sat at ~0.1 vCPU with the app unable to open new connections ("connection terminated due to connection timeout" from the pg driver). We saw the identical signature at 15:02–15:03 UTC yesterday, where ~30 backends' timeout cancels all landed within the same 300ms despite the statements starting at different times — meaning the cancels themselves couldn't be delivered, i.e. backends were stuck in uninterruptible waits (I/O), not busy executing.

▎

▎ Tonight's window started 6 minutes after your incident 8GL2R2U5 ("Deployments Slow to Start", root-caused as a networking issue on a single host) opened at 23:59 UTC, and recovery matched your 00:20 UTC fix. Can you confirm whether our Postgres service (project 5255cfae-7e88-41a9-b9bc-399c79e30b1e) was placed on or dependent on the affected host — and whether the same host explains the 15:02 UTC stall yesterday?

I have a ton of clients. Is there a official message? even the agents in railway chat box dont work

any updats what I can do i am tripping out

All my logs are bad on there just some 26-08-29 00:17:36.040 UTC [66601] ERROR: canceling statement due to lock timeout

2026-08-29 00:17:36.040 UTC [66601] CONTEXT: while updating tuple (818,23) in relation "power_dialer_sessions"

2026-08-29 00:17:40.083 UTC [66603] ERROR: canceling statement due to lock timeout

2026-08-29 00:17:40.083 UTC [66603] CONTEXT: while updating tuple (818,15) in relation "power_dialer_sessions"

2026-08-29 00:17:45.066 UTC [66604] ERROR: canceling statement due to lock timeout

2026-08-29 00:17:45.066 UTC [66604] CONTEXT: while updating tuple (818,15) in relation "power_dialer_sessions"

2026-08-29 00:17:50.072 UTC [66607] ERROR: canceling statement due to lock timeout

2026-08-29 00:17:50.072 UTC [66607] CONTEXT: while updating tuple (818,15) in relation "power_dialer_sessions"

2026-08-29 00:17:55.075 UTC [66608] ERROR: canceling statement due to lock timeout

2026-08-29 00:17:55.075 UTC [66608] CONTEXT: while updating tuple (818,15) in relation "power_dialer_sessions"

2026-08-29 00:18:00.023 UTC [66609] ERROR: canceling statement due to lock timeout

2026-08-29 00:18:00.023 UTC [66609] CONTEXT: while updating tuple (818,15) in relation "power_dialer_sessions"

2026-08-29 00:18:03.860 UTC [66484] ERROR: canceling statement due to statement timeout

2026-08-29 00:18:03.860 UTC [66553] ERROR: canceling statement due to statement timeout

2026-08-29 00:18:03.860 UTC [66567] ERROR: canceling statement due to statement timeout

2026-08-29 00:18:03.860 UTC [66573] ERROR: canceling statement due to statement timeout

2026-08-29 00:18:03.860 UTC [66552] ERROR: canceling statement due to statement timeout

2026-08-29 00:18:03.860 UTC [66536] ERROR: canceling statement due to statement timeout

2026-08-29 00:18:03.861 UTC [66535] ERROR: canceling statement due to statement timeout

2026-08-29 00:18:03.861 UTC [66581] ER

My MySQL instance is suffering, definitely a Railway issue.

But the current open incident doesn't seem to acknowledge the serverity.

Yes exactly the status at top does not mention its serious as my entire company and others is down!

💯

Incidents are only opened if it’s a widespread issue.

Try redeploying the instance.

i cant back up for 25 more minutes, can anything happen if I do before my daily back up?

Why is railway not posting new status update? What is ETA? This seems like a major outage.

@garrettmendelsohn @ricardomacario unfortunately seems like here is overlooked, for more urgent/immediate stuff is better you guys hop on Discord

What is the discord?

Im in the discord doesnt seem active either

I have seen some recovery, but the MySQL instance is still slow and unpredictable. Not back to 100% yet.

I got acknowledgement from the Railway team that they are looking into this.

Same issue, site down

I think this is more widespread than Railway knows - affecting production systems, not just deployments. Instability in Railway recently is a major cause for concern.

Same here…… why is this happening every 2 weeks? Railway is always have major outrages, it's unacceptable.

Same here, major network issues (working one minute, not the next) with old deployments across two separate services hosted in US West.

If you're experiencing degradation or connectivity issues, could you please reply to this message with a link to your database service? It'd help us trace the issue down. Thank you!

Are we all in us-west? add an emoji with a check ✅ if you're us-west and an X ❌ if you're not

i am yes

https://railway.com/project/5f8e54c6-b92b-47a8-82f8-9cc77e7a523b/service/67900c36-474f-445b-a679-99b32f70b0f1?environmentId=a42d6581-ecec-4826-aee5-884ebfcbbf85

I am but what do I send?

Thank you, looking into it

Click on your service -> paste the browser URL here

Here you go: https://railway.com/project/74d18d67-02d6-438e-b4f0-d8960d083b14/service/a408e8be-3181-4231-bd54-914044ef97c1?environmentId=71595b01-c5e8-4101-93cd-d730ddd90918

Postgres, us-west2, private networking. Degraded since ~23:45 UTC, still oscillating (last checkpoint 348s at 01:28). More diagnostics in my thread: https://station.railway.com/questions/severe-storage-i-o-degradation-on-postgr-7754040b

https://railway.com/project/5255cfae-7e88-41a9-b9bc-399c79e30b1e/logs?environmentId=52e02c05-a26e-480b-960c-dbd4cb36d261

https://railway.com/project/32bfb0d3-b1b4-4af3-ab7f-efdd0d2968cd/service/16f55d70-6e26-4b4f-99e4-5a551b96dc04/database?environmentId=9a28258f-28e2-42e7-bf38-73ff0fdbfaad

https://railway.com/project/3b7f49ad-5e67-4371-9170-a3baf8620d7e/service/939b3ace-390e-4cb5-bf9d-400c8d6737a0?environmentId=8a25d73b-2406-4ca5-8ea5-34874ea39f4c

Databases (""working"" but can't connect to them):

https://railway.com/project/8d610438-f424-474c-9f20-c0a84315f524/service/a7633bc9-cd3d-4c51-b56c-dc01b50825a9/database?environmentId=18c8b802-9a69-4346-8426-0d5628b31f05

https://railway.com/project/8d610438-f424-474c-9f20-c0a84315f524/service/a7633bc9-cd3d-4c51-b56c-dc01b50825a9/database?environmentId=17183ee2-2299-4cf8-a42b-eabf4159d484

https://railway.com/project/8d610438-f424-474c-9f20-c0a84315f524/service/a7633bc9-cd3d-4c51-b56c-dc01b50825a9/database?environmentId=98342170-27d3-4010-9e98-c6eebecc4c45

Backend (crashed and can't redeploy):

https://railway.com/project/8d610438-f424-474c-9f20-c0a84315f524/service/1c442682-8a90-4a38-94da-5ad5b2259ba4/database?environmentId=18c8b802-9a69-4346-8426-0d5628b31f05

https://railway.com/project/8d610438-f424-474c-9f20-c0a84315f524/service/1c442682-8a90-4a38-94da-5ad5b2259ba4/database?environmentId=98342170-27d3-4010-9e98-c6eebecc4c45

Can't connect to the database. https://railway.com/project/14b0db46-3742-4a7e-ad28-2b33c8eb2759/service/03339a9c-413a-470f-b2aa-eea676992db3/variables?environmentId=9f10ba65-ccd4-4827-bd6c-d33150c71ec3

Thank you everyone -- we've declared an incident at https://status.railway.com/incident/Z5Y3WO06 and we're actively working on resolving this. Really sorry for the disruption here!

This should be fully resolved now. Apologies for the disruption once again!