Prod Down due to networking connection problems
alfredbenoel
PROOP

4 months ago

We just faced this issue (as documented on https://station.railway.com/questions/networking-outbound-connection-problem-caafa330)

Angelo had told us to enabled static ip address and that this was due to intermeittment outbound-connectivity drop, and the pattern is always the same normal traffic flow -> 499 -> 502

Hi, is happening live to us right now, we have re-deployed 3 times, and it doesn't seem to be able to recover anymore.

Closed

23 Replies

alfredbenoel
PROOP

4 months ago


alfredbenoel
PROOP

4 months ago

@angelo (sorry for tagging you but this is happening live)


alfredbenoel
PROOP

4 months ago

The related deployments are: 97503f6e, 008df64d and 7862fc0b


hey there, all good, why I am taggable


working on this


so this is because of OOMing, not because of networking


cc @Lfret - are you around, it seems when it picks up a job from the queue its OOMing


are you using node for your backend?


alfredbenoel
PROOP

4 months ago

yes we are but looking at the timestamp (Jun 30, 2026, 12:50:58 PM GMT+8 Hash: 97503f6e), there were 499 errors way before there was the OOM crash.


thats also related to the OOMing is my guess


uh, you are logging bitstreams?


redeployed with an expanded memory heap flag for you if thats okay, monitoring


bump


alfredbenoel
PROOP

4 months ago

I think the bitstream is getting logged because our audio transcript job is failing, removing it now


okay, no crash


<:Fingers:1202140286956867644>


can you slam the backend for me?


alfredbenoel
PROOP

4 months ago

unfortunately we don't have any stress test tool yet


which tz are you in?


are, we are out of the crash loop yea?


alfredbenoel
PROOP

4 months ago

yes we are thank you so much angelo we will go and remove the OOM error now


alfredbenoel
PROOP

4 months ago

we are in the Singapore tz, thank you so much for helping us on this


np, least we can do


Welcome!

Sign in to your Railway account to join the conversation.

Loading...