Prod Down due to networking connection problems
alfredbenoel
PROOP

7 days ago

We just faced this issue (as documented on https://station.railway.com/questions/networking-outbound-connection-problem-caafa330)

Angelo had told us to enabled static ip address and that this was due to intermeittment outbound-connectivity drop, and the pattern is always the same normal traffic flow -> 499 -> 502

Hi, is happening live to us right now, we have re-deployed 3 times, and it doesn't seem to be able to recover anymore.

23 Replies


alfredbenoel
PROOP

7 days ago

@angelo (sorry for tagging you but this is happening live)


alfredbenoel
PROOP

7 days ago

The related deployments are: 97503f6e, 008df64d and 7862fc0b


hey there, all good, why I am taggable


working on this


so this is because of OOMing, not because of networking


cc @Lfret - are you around, it seems when it picks up a job from the queue its OOMing


are you using node for your backend?


alfredbenoel
PROOP

7 days ago

yes we are but looking at the timestamp (Jun 30, 2026, 12:50:58 PM GMT+8 Hash: 97503f6e), there were 499 errors way before there was the OOM crash.


thats also related to the OOMing is my guess


uh, you are logging bitstreams?


redeployed with an expanded memory heap flag for you if thats okay, monitoring



alfredbenoel
PROOP

7 days ago

I think the bitstream is getting logged because our audio transcript job is failing, removing it now


okay, no crash


<:Fingers:1202140286956867644>


can you slam the backend for me?


alfredbenoel
PROOP

7 days ago

unfortunately we don't have any stress test tool yet


which tz are you in?


are, we are out of the crash loop yea?


alfredbenoel
PROOP

7 days ago

yes we are thank you so much angelo we will go and remove the OOM error now


alfredbenoel
PROOP

7 days ago

we are in the Singapore tz, thank you so much for helping us on this


np, least we can do


Welcome!

Sign in to your Railway account to join the conversation.

Loading...