can you also provide the same data for the same app running on the legacy runtime

you mean change the runtime for that app, right?

I tested a different app running on Legacy and the issue didn't happen

but I'll change the runtime

testing a different app is not conclusive, when testing you need to change only one variable at a time, a completely different app changes too many variables

it was another Bun app, but I can see the variables

I'm deploying the reported app on Legacy and have to wait for it to sleep 🙂

I'm not talking about environment variables

yeah I meant variables as not in environment variables but how the apps are different despite both being Bun apis

deploy is done, will report back in about 10 mins

Test done, request on sleeping app worked fine:

HTTP/2 200
content-type: application/json;charset=utf-8
date: Thu, 04 Jul 2024 06:34:47 GMT
server: railway-edge
x-request-id: gLwN8r6fSpSLChJuzIS30g_3165824431
content-length: 15

{"status":"OK"}
real    0m1.795s
user    0m0.000s
sys     0m0.016s

switching back to V2 to repeat the test

btw that cold boot time = 🏆

1.795s is good?

for a cold boot time? I'd say excellent

I have another Rails app running on Railway that cold-boots in about 10s, kinda bad

but that's mostly Rails to blame

similarly I have a feeling this is bun to blame

I'm running a compiled executable, so very likely

isn't that the recommended way to run in production though

it is, I'm following their guide on that

for comparison, this is normal request time
```HTTP/2 200
content-type: application/json;charset=utf-8
date: Thu, 04 Jul 2024 06:38:27 GMT
server: railway-edge
x-request-id: vAXlgIj9Rlq8binHs09mAQ_603524580
content-length: 15

{"status":"OK"}
real 0m0.221s
user 0m0.000s
sys 0m0.000s
```

the increased memory usage on V2 is a bit of a bummer though

how much of an increase.

Legacy was running ~36MB
V2 is running 53~61MB

is this the exact same app, or are you comparing different apps again lol

it's the same app

someone else reported higher memory usage on the v2 runtime, but I can't reproduce it just by purely allocating bytes

just by looking at its memory metrics
looking further back (the app has been up only for a couple hours) the lowest it got on V2 was 42MB, but I only had one run of it on Legacy, so probably needs more data

but the difference is quite noticeable, maybe not in the image because the chart ceiling is a bit too high

btw the app went to sleep again and curl returned an empty response

if it matters, the service has a volume

remove the volume and try again?

I'll try, that might break the app though

app broke, trying to fix it

alright it's back up, now waiting for sleep

Network graph also wild on Legacy

then it's a good thing the legacy runtime will be phased out

got empty response on sleeping app, so volume is not it

okay can you provide a minimal reproducible bun app that sends an empty response on the v2 runtime

I'll try

I deployed a smaller app, and could not replicate the issue

but here's the thing… the affected app also stops having the issue 👀

uh.. task failed successfully?

ugh, technology these days 😛

ugh, bun these days

now that I have both apps running, I'll try to replicate it again with the affected app

then try to replicate it with the smaller app

code btw: https://github.com/pauldps/bun-railway-v2-test

I'll try deploying to separate project in case of same project shenanigans

that's definitely minimal

I was able to reproduce the issue with the minimal app in a separate project

and the original app also started showing blank responses after I removed the minimal app from that project 👀

this is looking more like instabilities with bun

try the same code with node?

added a node branch to the minimal app and deployed it, now waiting for sleep

just a question, why do you have the healthcheck timeout set to a low value like 30 seconds?

because I want it to fail fast
usually if the first request fails, the deploy is likely busted, and I don't want to wait 5 minutes for the deployment to fail

makes sense

I think for Rails apps with slower boot times I set a higher value

got the empty response with the Node app as well

interesting

can you link the applicable deployment

two requests

$ time curl -i https://bun-railway-v2-test-production.up.railway.app
HTTP/2 200
server: railway-edge
x-request-id: 3OBw1saOQ2WUAAJCR4VCGw_603524580
content-length: 0
date: Thu, 04 Jul 2024 08:17:05 GMT


real    0m1.322s
user    0m0.000s
sys     0m0.000s


$ time curl -i https://bun-railway-v2-test-production.up.railway.app
HTTP/2 200
content-type: application/json
date: Thu, 04 Jul 2024 08:17:29 GMT
server: railway-edge
x-request-id: 2SzMY7RyRVie9l27WpIy2Q_882434190
content-length: 19

{"status": "NODE"}

real    0m0.342s
user    0m0.000s
sys     0m0.000s

the deployment? or the project?

the deployment

oh, got it
7368d15e-ed13-4684-aab3-72e2b3bdaa74

full link please

the url in the browser when the deployment is open?

https://railway.app/project/46548220-e0ba-4a16-b80a-706a55133413/service/8da48915-af62-477c-b850-17671fee4e86?id=7368d15e-ed13-4684-aab3-72e2b3bdaa74

would it be too much to ask you to also do an express app?

lemme see if I can do it quickly, I never used express before lol

that's a crazy sentence, I had never imagined someone who uses bun and Elysia to say they've never used express

when express was a thing I was mostly working with Rails
when I moved to Node it was during a time where express was considered too slow compared to other libs, so I never touched it

express app is up

$ time curl -i https://bun-railway-v2-test-production.up.railway.app
HTTP/2 200
content-type: application/json; charset=utf-8
date: Thu, 04 Jul 2024 08:33:43 GMT
etag: W/"14-kjLmVQInBma0jJMTEoZwvPwAyY4"
server: railway-edge
x-powered-by: Express
x-request-id: F32SRDmRQFSyDPyKYfb06w_603524580
content-length: 20

{"status":"EXPRESS"}
real    0m0.388s
user    0m0.000s
sys     0m0.000s

waiting for sleep

code in the express branch

the express app responds correctly

you're still on the v2 runtime?

yes, I just changed the branch and nothing else

I wonder what's going on with Node's http server, which is probably what Bun servers are based off of

using express is not an option for me though

well that seems like this isn't an issue with railway then

hold on

got an empty response with express

I think my first test was too fast

it seems like an issue with V2 to me

could potentially test with non-Javascript frameworks but that would be a bit too much for me to do atm

ill test with a go server

what happens if i dont experience the same issue?

I can test with Ruby later, but for now I need to go sleep myself lol

good question, I have a theory, but want to test a slow language first

remember to deploy in a new project since it seems multiple services in a project can affect the results, I'd like to test more about that part too

i have indeed created a new project

I have deployed a Ruby/Sinatra app, and was not able to replicate the issue on the first cold boot. But I'm seeing a pattern in the logs that I want to investigate

these are the logs from the Express app. My first request did not trigger the problem, but my second did.

The second request was after the "container event container died" log entry that was absent from the first request. So I'm trying to get that log entry to show on the Sinatra app

the "Stopping Container" spam seems to indicate there's a problem somewhere with V2

was not able to replicate the issue with Ruby after 2 attempts. I'm going back to the main branch (Bun) to see if maybe the problem resolved itself

stopping container is it being put to sleep

does it show even if the app is already sleeping?

"Stopping Container" logs did not show up for the Ruby app 🤔

but it did go to sleep

(according to the dashboard)

maybe the ruby app is on the legacy runtime

I will doublecheck after I test one more time with the Bun branch

the problem is still there with the Bun app. The logs:

no "Stopping Container" tho

switching to the ruby branch for now to investigate more, made sure it's on V2

got to reproduce the issue with the Sinatra app. It was a little worse as two requests gave empty responses before the third one returned the correct response

$ time curl -i https://bun-railway-v2-test-production.up.railway.app
HTTP/2 200
server: railway-edge
x-request-id: OPED8JR1TW6SpSjny6blUg_882434190
content-length: 0
date: Thu, 04 Jul 2024 18:16:25 GMT


real    0m1.567s
user    0m0.016s
sys     0m0.000s



$ time curl -i https://bun-railway-v2-test-production.up.railway.app
HTTP/2 200
server: railway-edge
x-request-id: PbIYVlwJQ-qkE6uGJUtMsw_882434190
content-length: 0
date: Thu, 04 Jul 2024 18:16:29 GMT


real    0m0.211s
user    0m0.000s
sys     0m0.000s



$ time curl -i https://bun-railway-v2-test-production.up.railway.app
HTTP/2 200
content-type: application/json
date: Thu, 04 Jul 2024 18:16:35 GMT
server: railway-edge
server: WEBrick/1.8.1 (Ruby/3.2.4/2024-04-23)
x-content-type-options: nosniff
x-request-id: ZBY6eoJsS_qCym1oa23Yyg_3165824431
content-length: 20

{"status":"SINATRA"}
real    0m0.378s
user    0m0.000s
sys     0m0.000s

logs

(those are not errors btw, wtf Sinatra)

printed to stderr

I have requested my go app a few times when it has gone to sleep and was not able to get an empty response

were the logs like the above? I let my app sleep for about an hour or so before making requests

from observation the problem seems to be related to those "contained died" and "stopping container" errors

those are regular event logs, nothing to be concerned about

right, I meant logs, not errors 👍

yeah the container log stuff is perfectly normal

I do think they seem to indicate the container is going into a state where it fails to render responses on wakeup
so far V2 is the common denominator; I've changed projects and languages, and the problem doesn't happen on Legacy. What else could we try?

not sure, I'll report it to the team anyway

thanks, I'll keep the project up if the team wants to debug/investigate

So you can repro this on both bun and sinatra?

correct, also Node-http and express

Ack and escalated

It should be triaged on Monday

fairly certain that the blank response should in fact be a 503 application failed to respond page, but railway is no longer sending that page at the moment due to what i believe to be a bug.

so lets assume your first response to a sleeping service is a 503 status code, meaning your app did not respond to the first request in time, that explains why a statically compiled go app did not exhibit this behavior.

when a request comes in for a slept app the container is started and a tcp connection attempt is done on a loop every 30ms, once that succeeds the request is forwarded to your app, but if your app is not ready to handle http traffic just yet you will get the 503 app failed to respond page, the apps health check is not taken into account.

theres definitely some room for improvement here on the railway side of things for waking sleeping services aside from fixing the blank page being sent instead of 503.

Is this for the new proxy or?

yep all testing done with only the new proxy enabled

Great. Miguel merged a fix for this. Should be good to go

for clarity, the fix was for the blank response instead of the 503 application failed to respond page that should have been shown

I can handle the 503 response better than a blank page, can set my client to retry or something

(although it would be nice if the 503 didn't happen)

We now no longer return a 200

We should return a 500 as was the previous behavior

just ran a test now and got a 502 with a long HTML error page

That's correct ye?

that's better than a 200 for sure

What's "Best"

Best would be 200 with my app returning the correct response

Well yes

Is your app returning the correct response?

no, it's returning that huge HTML from Railway

No I mean like

that's just the first request though

What happens is this intermittent is it always

the next requests work fine

it's always when the app is sleeping

"first request when app is sleeping"

"First request when app sleeping results in 503"

Gotchu

Esclating again

This is only on the V2 runtime ye?

correct

does not happen on Legacy

I suspect it wasn't more widely noticed because it was returning a 200

Yep I suspect so too. Bubbled up

@Brody , is this application one of those app sleep healthcheck candidates ?

The fact the legacy proxy would work but new one does not makes me think the healthcheck feature we talked about wouldn't matter.

The new proxy is using the same timeouts as legacy and these edge proxies are not aware of application sleep logic. There must be a timeout setting that is different.

yep I feel like this could benefit from having the heath check checked on waking up the service

oh this is saying that this issue with app sleep waking only happens on v2 runtime but works on legacy runtime.

Correct ?

We just happen to be on the new proxy and saw those 200s (sorry about that…will work on ensuring issues that like that don't through)

I was able to reproduce and have a strong lead on the issue.

I'm continuing to work on this and will share updates here when I have some.

Hello, I'm still looking into this.

heads up, I saw a few reports of 502s and the people's containers were stopped

if you don't exit with non-0 the restart policy won't start it again.

unsure what stopped the people's applications but pointing out that checking the logs for container exits and restart the container resolved the issue.

noted, thanks for the heads up

I’m not sure if I’ll be digging in to the more since I found in 2 examples of this the container was stopped.

Next time someone brings this up I’ll just ask to restart their container to see if it comes back

I’ll have to look in to it if more people report of course

OP did say it was reproducible between multiple deployments

ok I don't think it's stopped containers in that case.

!remind me to reproduce this in 1 hour

BTW I don't think we have the cycles ATM to repro stuff for people

They'll have to come to us with reproductions that we can have a look at

In production I have an app with app sleep and v2 runtime. On my first request it says the app is unavailable. The container starts though. Second request it works.

So easy to reproduce.

Okay then givr

I don't understand this part: "the container was stopped". Do you mean the container was stopped manually? In my case the container stopped because it went to sleep. I didn't do anything to stop the container and my app is a long-running http server, so it doesn't stop on its own. The only reason it stops is the App Sleeping feature.

I can assure you applications can stop on their own

I think like, even if it does stop on it's own, we should probably restart it?

Cause like if it crashes and another request or something comes in IDK

if it exits with an error code, yes, if it exits with a success code, maybe not? but yes if restart is set to always

Brody got it for the restart behaviour

The fact I could reproduce this though means we can disregard what I said about the container being stopped.

I think mentioning it might have been a mistake as I was jumping between a few threads debugging stuff.

I will dig into this more next week.

hey folks, I spent some time on this and basically, the v2 runtime wakes and forwards http requests differently than v1 runtime. I have observed the success rate of starting and getting an HTTP response to be pretty flakey (sometimes it works, sometimes it does not). I believe this is something to do with how fast the container can start in v2 runtime before the request times out.

I can't spend more time on this right now because the number of reports for this has been small and have to prioritize some other issues.

If you need app sleep right now I advise just using the v1 (legacy) runtime.

another report of it here -

We’re expressly not going to be able to prioritize this until the new proxy is out unfortunately

Has something changed with this issue? My Legacy apps are starting to show 502 errors after coming back from sleep
Most of my apps also no longer allow me to change between Legacy and V2

We have indeed removed the ability for users to switch back to legacy

Is Legacy going to be removed soon?

when we move to metal legacy will not be supported, thus in the intrest of moving to metal faster all deploys for all plan tiers use runtime v2

I really need App Sleep to work reliably 😦

well then you will be pleased to know that the new proxy is indeed fully rolled out and 100% of the nearly half a million domains used on our platform now have their traffic served via the new proxy, thus we should be able to take a look at picking back up the 502 app sleeping issue.

That would be great, V2 working with App Sleep would be ideal

i've also bumped the linear ticket on your behalf

Appreciate that, thanks!

We will make sure this works reliably within the next 2 weeks

!remind me to circle back in 2 weeks

Just wanna say that we are actively working on a solution to this!