Repeated Out of Memory crashes
markojak
PROOP

5 months ago

Hi Team

Need help diagnosing what is causing the out of memory crashes on our API - Been pulling out our hair last 8 hours

Project: 7f34b4f7-9d09-48a8-997c-32a10e59c1eb

We have some errors from various functions but hard to pinpoint what is spiking the memory

29 Replies

samgordon
PRO

5 months ago

Is this a template? Written by yourself? Language?


markojak
PROOP

5 months ago

Besides the deployments logging is there anything else we can look at


markojak
PROOP

5 months ago

Not a template, it's our API


markojak
PROOP

5 months ago

It's an expressJS server


samgordon
PRO

5 months ago

Did you make a specific change and then it started occuring?


markojak
PROOP

5 months ago

we changed out our logger and it started happning around 7 hours ago


samgordon
PRO

5 months ago

What is the memory spiking to? How do you know it's an OOM?


markojak
PROOP

5 months ago

But we cannot determine anything further, everything points to us doing things more efficiently vs less efficiently


markojak
PROOP

5 months ago

repeated Railway OOM errors and crashes


samgordon
PRO

5 months ago

What's the limit you have set


markojak
PROOP

5 months ago

4vCPU and 6GB memory


markojak
PROOP

5 months ago

2 replicas


markojak
PROOP

5 months ago

Anything that we can do to get more detailed information on Railway?


markojak
PROOP

5 months ago

(thanks for the prompt reply btw)


samgordon
PRO

5 months ago

Railway won't profile your application


samgordon
PRO

5 months ago

I'm not sure how I would debug this on Javascript


markojak
PROOP

5 months ago

yeah


samgordon
PRO

5 months ago

Is this app open source?


markojak
PROOP

5 months ago

no private API sorry i guess there isn't anything else we can get from railway and need to look at DD etc


samgordon
PRO

5 months ago

Yeah it's not really Railway's responsibility here


markojak
PROOP

5 months ago

I agree


markojak
PROOP

5 months ago

Thanks for trying to help


Check for any added dependencies added to your project around the time the issue started happening


Run the Dockerfile locally to see if it's happening there too



markojak
PROOP

5 months ago

It’s a super hard problem because we think the errors were happening before but the OOM started after a huge logger refactor

1414623786946531600


markojak
PROOP

5 months ago

Going to try out the heap snapshot thanks @astrid(irl)*


samgordon
PRO

5 months ago

That's what you get for blindly trusing AI…


Can also git bisect


Loading...