API Crashes 20+ Times Daily
jared-leddy
HOBBYOP

a year ago

Trying to figure out what is going on. Our uptime monitor sends out an email when the page goes down for longer than 5 min. For the past few weeks, the API is showing that it goes down 20-40 times a day for about a 3 day stretch. Then it's fine for a few days, and it's back to a 3 days stretch of chaos.

What?
How?
Why?

34 Replies

brody
EMPLOYEE

a year ago

please provide more information, for starters, are there any error logs?


jared-leddy
HOBBYOP

a year ago

There are plenty of info logs, but they don't have any details on them that indicate an issue. There are no error, debug or warn logs for the past 2 months.


brody
EMPLOYEE

a year ago

I would recommend adding some very verbose debug logging sonyoi can determine at what point your code crashes


jared-leddy
HOBBYOP

a year ago

It's built on Nest.js and the error/warning logs are typically pretty solid. I can look at adding something else, but that may take a bit.


brody
EMPLOYEE

a year ago

railway isn't going to have the observability into your app if your app doesn't have the observability you need to determine the issue


jared-leddy
HOBBYOP

a year ago

In English, you're saying that the logs in Railway are only as good as the ones built into the app.


brody
EMPLOYEE

a year ago

that's correct


brody
EMPLOYEE

a year ago

if you don't know why your app is crashing, railway isn't going to know either


brody
EMPLOYEE

a year ago

besides things like OOM but that's easy enough to determine from your side


jared-leddy
HOBBYOP

a year ago

That's the problem I believe. We're using Nest instead of Express in part because of the built-in logs. If the app actually crashes, Nest will let you know. But I'm not seeing any logs that say that the app actually crashed.


jared-leddy
HOBBYOP

a year ago

If the app never actually crashed, then the app has a problem with 1 page going haywire, or the response time is too long.


jared-leddy
HOBBYOP

a year ago

The uptime monitor is showing that the API went down 7 times on 2024-06-21 for an estimated total 7 minutes.


jared-leddy
HOBBYOP

a year ago

It doesn't actually tell me why, but this is a "keyword found" type monitor.

1254584995037581300


jared-leddy
HOBBYOP

a year ago

This monitor is an HTTP ping. It shows nothing happening on that date.

1254585224185119000


brody
EMPLOYEE

a year ago

I'm sure there's a hundred or more ways your app could crash or soft lock without nest knowing.

are you on the v2 runtime? and on the new edge proxy?


jared-leddy
HOBBYOP

a year ago

I suspect that is probably true, though the how that can happen seem lost on me. I'm guessing no on v2 and edge, as I don't know what those are.


brody
EMPLOYEE

a year ago

check your service settings


jared-leddy
HOBBYOP

a year ago

My Railway settings say Legacy runtime.


brody
EMPLOYEE

a year ago

and the edge proxy?


jared-leddy
HOBBYOP

a year ago

Not enabled.


brody
EMPLOYEE

a year ago

does your service have a volume?


jared-leddy
HOBBYOP

a year ago

I don't think so. I can't find anything that says Volume in the settings.


brody
EMPLOYEE

a year ago

it's not in the settings, look at the project canvas


jared-leddy
HOBBYOP

a year ago

I'm guessing this is the canvas. If so, then it's just a Github repo and PostgreSQL DB.

1254669661874688000


brody
EMPLOYEE

a year ago

there's no volume on the API service


brody
EMPLOYEE

a year ago

you can see the postgres service has a volume


brody
EMPLOYEE

a year ago

enable the v2 runtime and edge proxy on your API service


jared-leddy
HOBBYOP

a year ago

I see what you're talking about. The bottom box.


jared-leddy
HOBBYOP

a year ago

Deploying the updates now.


jared-leddy
HOBBYOP

a year ago

That's done.


brody
EMPLOYEE

a year ago

okay continue monitoring the service and report back


jared-leddy
HOBBYOP

a year ago

Copy.


ayush-lal
HOBBY

a year ago

i've also been having an issue with my nestjs API restarting sporatically throughout the day. Havent had time to look into it though, i believe i used one of the existing railway templates. Did you also use the template @jared.leddy?


jared-leddy
HOBBYOP

a year ago

No, we didn't do anything fancy. Just connect the repo and quick deploy it with ENVs and a DB.


Loading...