US East, super slow
coderjoshdk
TRIALOP

2 years ago

I just moved "everything" over to US East. And …. it was slow. Like broken levels of slow. The requests were all delayed by seconds. And s would simply not work (or take close to 10 seconds to load).

For now, I moved everything back to US West. And it is fast again.

I put everything in "" because there was one thing not in US East. the DB was in West. This could be something that slows it down. But no, this was seconds of delay between requests. Requests that do not take long at all to make.

I really wish I could sit here and help debug but I can't afford to have prod be in a broken state for my testing purposes. I might test again and put everything in US East on off working hours (IE when my servers won't be used). And while I do that, I will test with the DB in US West and then move it over. But beyond that, idk how much testing I will be able to do to help debug this issue.

I don't know what the issue actually is though. Becuase it isn't like like request are taking a long time to respond based on the request durations (both client and server were claiming quick responses. Lies!!!!)

project id: 1b326884-0c17-43ed-9b52-f443662e8f50

45 Replies

coderjoshdk
TRIALOP

2 years ago

Sorry for breaking things again <:cry_gil:795020958352146443>


coderjoshdk
TRIALOP

2 years ago

I wish I took a video like I normally do. But Like I said, I was on pure "fix prod" mode. So getting a video to debug the platform was not on my mind


coderjoshdk
TRIALOP

2 years ago

<:angrykirbs:786254827243962390> and of course I didn't have my analytics tool on that would normally have picked up this recording. Welp, wish I could provide more data. Really want to know what happened. But oh well I guess. ….


brody
EMPLOYEE

2 years ago

qq where you connecting to the database privately?


coderjoshdk
TRIALOP

2 years ago

The backend was, yes


coderjoshdk
TRIALOP

2 years ago

And while I could see that messing up the backend, I am not convinced it would mess with the frontend so much. In theory possible. But I have a hard time seeing that be the case. I will investigate like I said. But either way ….


brody
EMPLOYEE

2 years ago

so would static assets take up to 10 seconds to load too?


coderjoshdk
TRIALOP

2 years ago

Yea


coderjoshdk
TRIALOP

2 years ago

No, its ok. Its ok to laugh.


brody
EMPLOYEE

2 years ago

thats not me


brody
EMPLOYEE

2 years ago

i promise


brody
EMPLOYEE

2 years ago

i assume you have the edge proxy enabled on the frontend service right?


coderjoshdk
TRIALOP

2 years ago

To remind you on the structure of this project; the frontend and backend are privately talking to a reverse proxy that is exposing itself. So that reverse proxy did have the edge proxy enabled. It was actually one of the first things I tried; toggling between it on and off

1252468186260181000


brody
EMPLOYEE

2 years ago

did the proxy service have the edge proxy enabled the moment you move the region of it?


coderjoshdk
TRIALOP

2 years ago

I honestly don't remember if it was enabled before or after moving regions.


brody
EMPLOYEE

2 years ago

may i ask your region


coderjoshdk
TRIALOP

2 years ago

US East


coderjoshdk
TRIALOP

2 years ago

Hence me wanting it on east


brody
EMPLOYEE

2 years ago

this sounds eerily similar to what another user reported



brody
EMPLOYEE

2 years ago

they arent pro, so it was just after first deploy to us-west


brody
EMPLOYEE

2 years ago

but i dont know what me or the team could do here tbh


coderjoshdk
TRIALOP

2 years ago

Not really asking you to do anything. This is more like a pure, throw hands up in the air, bug report.


coderjoshdk
TRIALOP

2 years ago

I really love the tools and ideas of railway. But overall it has been a very buggy experience. It is a lot of little things. Most all of the bugs can be excused and don't bother me at all. And I will still be using it for this project. But I am not convinced on migrating other projects onto the platform. The little things do add up.
And I am not saying this to make anyone feel bad. Actually, if anything, it is the opposite. Despite the buggy experiences I have had, I am still really interested in using railway. And I really do want to see this company / products grow and succeed. I am just not sure I will be putting any project that requires pro status on the platform just yet. But even at that, I am sure I will migrate some of the projects over here.


coderjoshdk
TRIALOP

2 years ago

Examples of "the little things"

  • Groups sometimes just decide to bunch up (pic 1 & 2)

  • Randomly the screen will flash with 404 and then everything will return to normal

  • Sometimes views just don't load (like if I click on "logs" on a service) so I have to escape and then do it again. And then it will load

  • All the bugs I have been pointing out in my other posts.

  • Some other little things I hit with root directory and adding a repo (will likely add details about that in a different thread)

Again, despite that, I still do like the platform. I like the ease of use. I like the way it aggregates logs. I like the the observability. I like the auto deployment on push and ease of environments. I like the structure of projects and services. I like a lot of things. But I am also a little frustrated. I am sure I will get right back to it though after I get some sleep. So I am going to go do that now

1252475055645130800
1252475055976222700


brody
EMPLOYEE

2 years ago

i feel you, ive been here with you and i can tell you are a tad frustrated, and i can't blame you one single bit, you have all the right to be because you for sure have not had the best experience with the platform, if you have the time would it be possible to ask for a list of all the threads you've opened so that the team has all them brought together into a single place?


coderjoshdk
TRIALOP

2 years ago

Yea I can get that all together. Tomorrow is either going to be super busy or super not. No in between. So if I get it tomorrow, it was super not busy.


brody
EMPLOYEE

2 years ago

no worries, and thank you for your tremendous patience despite your poor experience with the platform


jake
EMPLOYEE

2 years ago

I put everything in "" because there was one thing not in US East. the DB was in West.

If you're doing something weird with your querying, 100% it could blow up like this


jake
EMPLOYEE

2 years ago

seconds would have to be something thats all kinds of messed up, but, this is the first I'm hearing of this here

And I think that the conductors are? We run blackbox testing with latency pings and they look fine


brody
EMPLOYEE

2 years ago

that would be a safe bet to think it was backend to database issues, but even static assets from their frontend where slow to load



adam
MODERATOR

2 years ago

uh oh us-west is broke


adam
MODERATOR

2 years ago

hope this isn’t a predictor of an incident


adam
MODERATOR

2 years ago

1252493812182290400


adam
MODERATOR

2 years ago

hmm


adam
MODERATOR

2 years ago

intermittent failures, I refreshed a couple times and ~2/3 of the time it came back with this


coderjoshdk
TRIALOP

2 years ago

I should be sleeping; but ummm. I am not. Anyways, can confirm. I see the same thing


adam
MODERATOR

2 years ago

As should Brody and I lol


brody
EMPLOYEE

2 years ago

well it is SSR so anyone would see the same


coderjoshdk
TRIALOP

2 years ago

I really really don't think so. It again could be the db. However, yea, the static assets would take a long time. But also, the client and server were claiming that the response duration were ms. But it was definitely not. So something strange with something was going on. Maybe it was the private network. Maybe it was a fluke failed/messed up deploy. Idk.
Oh and also yea, I don't use a VPN


jake
EMPLOYEE

2 years ago

Think I need to redeploy that one. But if you check any of the other ones, they’re rock solid


coderjoshdk
TRIALOP

2 years ago


jake
EMPLOYEE

2 years ago

Maybe? Has it resolved itself


coderjoshdk
TRIALOP

2 years ago

idk. I moved everything back to west. I might test moving things over to east later today. We shall see.


Loading...