Seeing dramatic changes in response time
malachite40
PROOP

10 days ago

Seeing dramatic changes in response time.

I don't really understand whats happening. I deployed a few console logs, and my response times jumped to 6-8 seconds p99/p50.

I redeploy, the issue is gone. What in the world is going on?

$20 Bounty

4 Replies

Railway
BOT

10 days ago

This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.

Status changed to Open Railway 10 days ago


chandrakiran-gr
HOBBYTop 10% Contributor

10 days ago

What runtime are you on? If it's Node, this feels like the console.logs blocking the event loop.

Notice your P50 moved, not just P99. If the median request is 6-8s, basically everything slowed down, which usually isn't app logic or the DB. console.log writes to stdout synchronously, and in a container stdout is a pipe into Railway's log collector. Add more log lines, and each request spends more time blocked on those writes. If the collector drains even slightly slow, the pipe buffer backs up and requests to wait on it. A redeploy clears it because the new container starts with an empty buffer until it fills again.

Quick way to check: pull out the logs you just added and see if it goes back to normal without redeploying. If it does, that's your answer. Watching event-loop lag (perf_hooks monitorEventLoopDelay or your APM) will also show spikes lining up with the latency, while CPU/memory look fine.

Fix is to not log synchronously on the request path. pino with a worker transport is the usual move, and keep the per-request log objects small.


malachite40
PROOP

10 days ago

Hey! Thanks for the response. I am on node / nextjs 16 + postgresql container

I’ve tried removing logs, adding logs, changing queries, and adjusting connection pool limits, but there doesn’t seem to be any correlation.

Sometimes adding logs makes the issue disappear for that deployment; other times, removing them causes it to return. Rebuilding the exact same container may resolve the slow response times until the next deployment, while another rebuild can reintroduce the issue.

At this point, I strongly suspect the problem is unrelated to my code and may instead be tied to the underlying Railway hardware or infrastructure.


malachite40

Hey! Thanks for the response. I am on node / nextjs 16 + postgresql container I’ve tried removing logs, adding logs, changing queries, and adjusting connection pool limits, but there doesn’t seem to be any correlation. Sometimes adding logs makes the issue disappear for that deployment; other times, removing them causes it to return. Rebuilding the exact same container may resolve the slow response times until the next deployment, while another rebuild can reintroduce the issue. At this point, I strongly suspect the problem is unrelated to my code and may instead be tied to the underlying Railway hardware or infrastructure.

chandrakiran-gr
HOBBYTop 10% Contributor

10 days ago

Yeah, that pattern basically rules out your code. Same container, different speed per deploy, no correlation with logs or queries, fixed by rebuilding until you get lucky. That's host contention, a noisy neighbor on whatever shared machine you landed on.

Railway packs containers onto shared hosts, and each deploy can reschedule you onto a different one. Land on a busy box, and every request slows down; the next rebuild puts you somewhere quieter. Nothing in the app moves it, which is what you're seeing.

What confirms it is CPU steal time, the time your vCPU was ready, but the host was running someone else. On a slow deploy, you'll see steal elevated while CPU and memory look fine. (If it's high I/O wait instead of steal, same idea, but storage contention.)

Not much to do app-side except redeploy until you land well. Worth flagging to Railway with timestamps of a slow vs fast deploy, and for a steady production workload, a dedicated CPU is the real fix.


malachite40
PROOP

10 days ago

How does one tag railway to actually look at a ticket?


Welcome!

Sign in to your Railway account to join the conversation.

Loading...