9 months ago
I've had a sveltekit / laravel deployment up for a week in production without issues. Out of nowhere today, data fetching has become so slow that I had to increase timeouts from 10 to 25 seconds just so that the data wouldn't time out. It's been incredibly frustrating.
I have the identical code running in a staging environment and my local also uses staging. These systems have been working fast and fine for this past week too. I literally cannot figure out why this would be the case? What's happening?
25 Replies
I've tried switching up servers, locations, redeploying all code inside the production environment.. even reverted to git branches from weeks ago to see if it's anything related to code.
But it can't be the code. Staging runs perfectly fine and fast. So does my local.
What happened today some hours ago is that my postgresql in production was migrated to metal
project: cdf1815d-d3de-4820-866b-7fea03248e92
service: 6fc7d6e6-2ed1-4887-8a2a-7fd8d8722cd9
env: c78560a7-98ec-4264-9b77-000a99c91588
I have switched the frontend back to metal and still, it's just extremely slow. staging works flawlessly. I haven't made code changes in over a week
I'm migrating it to metal servers along with everything else to hopefully have it back up to speed
9 months ago
Hey, I have a simillar issue - (though using Django backend with postgres and NextJS front). All in the same region, just a few days ago the delay on write requests is slow. For example when saving or creating a new object it takes a few seconds for the request to come back, if I repeat a simillar request right after it works at normal fast speed.
Started happening just a few days ago
Similar issue with a Django backend and Postgres database. Write queries can be extremely slow and cause timeouts after 30+ seconds. Started just a few days ago.
@fred my issue was that railway migrated the postgresql database to another location and my laravel backend stayed where it was. Deploying my laravel backend to the same server region fixed this.
Imo this is a major oversight on railway @Percy
@fred even if same Region, try moving both your deployments somewhere else, then back onto the same metal server of your choice.
9 months ago
please do not ping the team. if you have feedback, post it on the station or #🤗｜feedback
I'll be writing my own help thread for sure, just quickly commenting here to confirm what fred said that initial WRITE database query seems to have become 2 orders of magnitude slower on Metal a few days ago (during the weekend I think, didn't have monitoring in place…) without any changes to the application data access patterns. But I don't want to cry wolf yet without being sure I didn't cause that.
had a similar issue on us west metal. starting a few days ago without warning my postgres requests would spike and take like 20+ seconds to complete. it appeared to only happen when writing to the db not reading from the db and not all the time it was inconsistent. i fixed by migrating over to us east metal which seemingly resolved the problems immediately
9 months ago
please share project/service ids and timestamps everyone. without specific details, the team can’t identify what’s happening.
My problem is hopefully being worked on - you can read about it here https://station.railway.com/questions/assistance-requested-postgre-sql-sudden-04258f1a
I encourage all other people who commented in this thread to write their own help threads similar to mine. Your reports will help the Railway team prioritize.
9 months ago
!s
Status changed to Solved brody • 9 months ago