Intermittent high response times I can't debug.
nerdvananc
FREEOP

2 months ago

I'm having some trouble with high latency on my app, and I can't seem to debug it using the information I've found so far. Especially because my app runs just fine at times, and then suddenly starts being SUPER SLOW, even when I am on the same page and don't do anything differently.

My app runs on Flask and uses Flask-SQLAlchemy as the ORM and Railway Postgres as the DB. There are no frontend libraries involved.

Also, my app doesn't even have a single user yet - it's just me with two test users I created, and there's a total of <20 records in the DB at this point. I have a strong feeling that it is not because of the database, because these issues existed on pages that did not involve any database queries before I cached them through Cloudflare.

Furthermore:

  1. I've made sure I'm connecting to Postgres privately by using the DATABASE_URL variable, not DATABASE_PUBLIC_URL

  2. I have all my services deployed in the same region: `US West (California)`

  3. I'm from India, but very often I'll see x-railway-edge be in EUR, not SEA.

  4. In my HTTP logs, almost every request is more than 500ms, even those that don't involve the DB.

I've attached these screenshots to this post:

  1. One showing a subset of the HTTP logs from Railway.

  2. One showing graphs from the Monitor tab from the service.

  3. One showing a subset of the Postgres instance's logs.

  4. One showing the timings of an especially painful request from Firefox's Network inspector.

Any ideas are welcome, thank you!

Solved

4 Replies

nerdvananc
FREEOP

2 months ago

Since there are no replies thus far, I did some more research.

I timed my requests' processing times from within Flask (using @app.before_request and @app.after_request) and compared them to Railway's HTTP logs.

On average, Railway itself is adding anything from 500~900ms to every single request. Bringing the total to an average of >650ms. I don't think this is acceptable, because I've seen Wordpress sites in production load faster on PHP/nginx servers.

After all of that, there's also times (not cold starts, happens in b/w active sessions) when this shoots up inexplicably. Unfortunately I wasn't able to figure that part out; but ALSO keep in mind the fact that x-railway-edge often messes up, the end result hovers around 0.7~1.1s for normal requests and even >5 SECONDS for more painful requests which have nothing to do with my code being unoptimized/slow. worried emoji

I have attached my logs below as photos for reference. I'm hoping that this is an easy solve, maybe I should upgrade? Or I've just overlooked something simple? Please assist because I love using Railway. :)


2 months ago

You have your service and database in us-west, yet most of your traffic is not originating from us-west, I would recommend moving your services to the region closest to where you will be accessing them.


Status changed to Awaiting User Response Railway 2 months ago


brody

You have your service and database in us-west, yet most of your traffic is not originating from us-west, I would recommend moving your services to the region closest to where you will be accessing them.

nerdvananc
FREEOP

2 months ago

Hi Brody, thank you for your reply. I don't think this answers why I'm seeing 5s+ response times on random requests, but I understand. Thank you for your help.


Status changed to Awaiting Railway Response Railway 2 months ago


2 months ago

Once you move let us know what it looks like! Would like to see the latency grin emoji


Status changed to Awaiting User Response Railway 2 months ago


Railway
BOT

2 months ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway about 2 months ago


Loading...