Hey there! We've found the following might help you get unblocked faster:

• 🧵 Really slow PostgreSQL queries…

• 🧵 SLOW QUERIES

• 🧵 Dashboard queries slow or failing

If you find the answer from one of these, please let us know by solving the thread!

Hey there Mark,

Unfortunately this is a known issue, we're trying to investigate the root cause of this regression. Do you have some logs or traces when the spikes occurred with the slow reads/writes?

Hi Angelo, I can see the spike in API latency in Sentry. It's hard to get Sentry to label the graph nicely, those previous spikes are back on July 14th, then July 9th.
There doesn't seem to be a correlation with server activitiy. The site isn't especially busy today, we're not curently running anything like billing runs etc.

Is it possible to revert back to GCP till the issue is better understood? It's really hurting performance.

That was the next thing I was going to suggest, I have enabled the ability to migrate back for your account. You should see a dropdown with options that say: "Legacy"

Thanks, currently though I see two different EU Wests but no legacy label? I've tried refreshing the dashboard.

I don't see anything in account settings.

Hi Angelo, any update on this? Whilst our API/postgres latency has settled for now, we would still like to move to GCP until the underlying issue can be identified. Obviously happy to help with any investigation.

Jeez, that must be frustrating. We merged a PR that removed the labels, going to ping the team.

Hi Angelo, before I open another thread, I'll write here. We have been having the same problems for some time. Very simple queries often take 3-5 seconds and sometimes up to 12 seconds. I first thought it was due to our application, but I run simple UPDATE methods directly on the database and these also take 3-5 seconds. I have the feeling that this is the case since we switched to Metal. Though I have a feeling it's only in the prod environment. Is there anything we/you can change or do I need to create a separate thread for this?

Heya, we'll need you to make a separate thread

Ok thanks, I have opened a new thread. https://station.railway.com/questions/very-slow-queries-on-postgresql-database-886b9b0f

🛠️ The ticket Performance issue with disk operations on metal has been marked as todo.