21b7eb49-e84c-40c9-9278-73e17ff7b318

For context, railway does ddos protection for themselves, not for an individual users sake.

a 503 simply means your app failed to respond properly to the request.

Yup, that's what I expect. The only issue is I'm not seeing any indication of an issue on my server <:thinkies:1238577143466692739>

It's a pretty standard express server, it's doing some stuff but I'd expect those things to have some indication when they fail. My suspicion is the request isn't even getting to my server

can you reproduce this with something better than express?

Debugging a prod issue so a smaller repro may be on the backburner. Understand if this means this help thread is lower prio 🫡

ticket?

Meant this thread 🫡

there's unfortunately not much I can do for you here, as hobby users are only eligible for community support

Gm, sorry to revive an old thread but we're still seeing this issue, and across multiple of our services we run on railway.

tldr is very sporadic 503s. They only seem to repro under load, so it's hard to have a consistent repro. But we've received enough reports from different external parties across many of our services to suspect it's related to railway, or we're doing something else wrong.

I'm seeing this issue across apps that are running on both express and next.js. I did some digging in past help threads and looks like there are a decent number of threads with this type of issue:

• https://discord.com/channels/713503345364697088/1148157220777885757

• https://help.railway.app/questions/occasionally-getting-503-responses-expr-57b386b5

• https://help.railway.app/questions/deployed-app-responds-randomly-with-503s-07e53c87
  In one of the threads there's strong confidence built that the 502 at least seems to be a payload something along railway's side is returning, and in the last one it's suggested for users to run their own proxy next to externally exposed services. Given that I'm not seeing any error logs across my services, I wanted to see if there are any other suggested workarounds?

Also, I'm on the "Pro Plan", I just have my discord linked to my personal railway account vs my "work" one. Happy to redo that link if it's useful. 😄

Appreciate any advice on this nefarious issue <:salute:1137099685417451530>

do you currently utilize cloudflare?

Yes

have you tried switching to cloudflared?

have you tried switching to cloudflared?
No I haven't, from a quick look it seems relevant, thanks for the pointer. Have you used it in the past with railway with success?

I made a dead-simple hono web server and am observing the same behavior. If I ramp up my "load test" to ~200 reqs per second, I get pretty consistent failures after not too long, with no indication of errors on my server-side. One hypothesis I have is that there's an issue with the server scaling up quickly to handle load; the metrics on my server show it barely raising its consumption, despite the start of the 503 errors. I'd expect the memory/cpu to creep up and hit some limit before the 503s creep in.

if i gave you an endpoint to test, would you be able to test it?

My test script is very simple:

const honoUrl = "https://hono-test.up.railway.app/";

// ~200 reqs per second.
const concurrency = 100;
const delayMs = 500;

let errors = 0;
let numReqs = 0;

const delay = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
const makeTestRequests = async () => {
    try {
        const response = await fetch(honoUrl);
        const data = await response.json();
    } catch {
        errors++;
    }
    numReqs++;
};

while (true) {
    const start = Date.now();
    // Make a bunch of requests concurrently.
    await Promise.all(
        Array.from({ length: concurrency }, () => makeTestRequests()),
    );
    const now = Date.now();
    const elapsed = now - start;
    console.table({ errors, numReqs, elapsed, now });
    // Wait before going again.
    await delay(delayMs);
}

export type {};

Have you used it in the past with railway with success?
yep! set it up for a user a while ago and they have yet to tell me about any issues, but it wasnt like they where having issue with railway to begin with

My hono server is dead-simple too:

import { Hono } from "hono";

const app = new Hono();

app.get("/", (c) => {
    return c.json({ now: Date.now() });
});

export default {
    fetch: app.fetch,
    port: process.env.PORT || 3000,
};

okay, one minute, ill have an endpoint for you to test

I can also try standing up the cloudflared if you have a quick tldr guide 👀

i dont, but i can walk you through it after my dinner

Just stepped afk, but assuming this will be up later will give it a test tonight <:salute:1137099685417451530>

it wasnt up when I sent it?

It was up, I just stepped afk briefly

I'm running my script against it now, I get a pretty high number of failures <:thinkies:1238577143466692739>

oh im very dumb

For reference, here's my output my endpoint

(still higher error rate than I'd want)

theres a rate limiter

Oh lol, lmk if you can disable and I can try again

Btw one Q for your cloudflared suggestion, would this mean I have to pay for egress to both railway and cloudflare? I haven't used cloudflare that deeply aside from just managing my domain/dns

you would have to pay egress for the cloudflared service you deploy to railway, and i dont think cloudflare charges for egress?

its a rate limiter for every endpoint in the app, so ill have to do research on how to disable it for a single endpoint

figured it out, building now

i guess im not building

okay changes are live now

feel free to run the test again

how did the test go?

@sina sorry for the ping, but the new edge proxy just went into beta and id like to see if enabling that magically fixes the errors you see when doing your test

it will take railway's old and slow envoy proxy out of the mix and replace it with a fully custom in house built proxy

Amazing, sorry ser I'm still tracking this just afk for the weekend. Took a peep at my settings though and it doesn't let me switch the toggle on?

haha its a little unresponsible, i had to hover it for several seconds before it let me interact

Nvm the toggle is bugged but looks like it worked!

Yup

Switched it on, will monitor. Was planning to run more tests next week

Appreciate you ser 😁

you did run more tests on my endpoint after i removed the rate limit on the now route, did you get any errors?

No errors from yours but only did limited testing, want to do some more.

Played with the new edge proxy and it seems my local testing quickly hits a 429: Too many requests error! Curious if this is documented anywhere so I can understand how it works (guessing an IP-based sanity rate limit, would be good to understand though)

okay so its not strictly an issue with railway

since its in beta, theres no docs around it yet, i will tag in mig (the person who spent a year developing it) here monday for extra insight

I have a vanilla express service that also returns just the simple date and it was having pretty frequent 503 errors crop up. I wanted to do more testing before drawing any conclusions 🙏 🫡

Glad to get to beta test it 😁 I'm very optimistic

how many req/s did you say your test does?

This config errors out pretty quickly. It's not perfect reqs per sec because I'm using my own hacky script, but basically I fire off 100 requests and after they finish wait an additional 200ms before going again.

So not accounting for the request time, this is ~500req/sec

Was considering spending more time min/maxing it, but may wait until Monday if there's a chance we can just get shortcut'd to the right answer via the inside word

500 req/s is a lot, but its a bit too soon for railway to be returning 429 imo

will see what mig has to say

I just tried again at 100 req/sec and errored quickly again

Yup, I'm not too concerned <:salute:1137099685417451530>

okay we'll come back to this monday tuesday
(monday was a holiday)

Gmgm, wanted to ping this thread, eager to hear more about the new edge proxy 😁

new load balancer? the changelog only mentioned that as a down the road possiblity, nothing has been implemented yet

I corrected my message to edge proxy*, sorry for the mix up, early morning 😆

ah gotcha

I have already told mig about it, and linked this thread to him, waiting to hear back

Appreciate it ser 😁

Hitting 429 rate limit responses @ <20 reqs/sec as well (which seems quite low…)

still waiting to hear back from mig

hey @sina sorry for the delay, some changes where made and you should now be able to do a max of 1k RPS on the new proxy

Amazing, will continue my testing then, thanks 😁

Had the following occur today:

• service has two URLs assigned to it-- one up.railway.app one and one via cloudflare

• it redeployed via github push

• newly deployed service didn't serve on the cloudflare URL, but showed up fine the up.railway.app one

• attempted to clear cloudflare cache and redeploy; finally seems to be fixed after I disabled the new edge proxy

Going to play with flipping the edge proxy back on tomorrow, but wanted to flag.

Here's a failed request ID for one that was failing to load via the cloudflare URL while the up.railway URL was working: GAhb00FYS6SLQefqUm3Aeg_1861343781

what was the reason for the failure?

No clue, that's why I pinged the thread 😅

the page would have told you

Played with re-enabling the new edge proxy and looks like it makes the services return 404, so seems like that suddenly broke for some of my services

Disable edge proxy -> domain loads fine (after waiting a few min)

please be more specific when you say "didn't serve"

The above 404 screen is served^, and I don't see any corresponding error log or crash on my service. Let me know if I can be more specific 😁

After disabling the edge proxy, looks like the [up.railway.app](up.railway.app) domain also continues to serve via the edge proxy (can tell because it's showing me the edge proxy's 404 page, the old proxy's is different).

The custom non-railway domain works fine and serves my normal app though..

DNS would take a bit to update, I'll see if I can reproduce this

I'm not seeing this issue for other services, but also now I'm a bit worried to redeploy and play with it in my other services 😅

Let me know if I can provide project id or anything for help debugging, seems like an edge proxy thing to me.

Fwiw, here's the settings; the [railway.app](railway.app) URL 404s (showing the edge proxy's error screen) and the cloudflare one works fine. This is a dev instance so happy to flip the settings around if it's helpful.

sorry but this thread is only going to be for issues relating to the edge proxy but you do not currently have it enabled

My apologies if I'm being confusing, but I'm trying to describe general strange behavior with the edge proxy. In the case of the screenshot, I'm still getting the edge proxy's 404 page despite disabling it ~20 minutes ago.

But enabling it also breaks my custom domain from working, I think that's the primary issue. So when I enable it, the [railway.app](railway.app) domain works but the cloudflare one breaks.

Maybe I just need to give more time for ~dns things to happen; will enable it and check back in the morning.

please keep the edge proxy on

Err, I have it off for essential services since they aren't working with it on 😅

Btw check-in for this morning, the dev service that I flipped it on for is still in this state

please be more specific when you say "aren't working"

Hey, I reply-messaged the earlier context that should specify. If there is something specific that I'm not providing that would useful please let me know, but asking me to re-supply the same context over and over again makes the thread pretty noisy 😁

It's the same error screen I'm been mentioning since my first ping yesterday

This is with the edge proxy ON since last night.

Again, the [railway.app](railway.app) URL works, the custom cloudflare one does not.

I have a duplicate of the same service which has its custom domain working with the edge proxy disabled, so between that and the 404 page being the edge proxy's (again, the screenshot of the not found page!!) I'm assuming the issue has to do with the edge proxy

please please don't worry about making the thread noisy, adding a tad bit of context in place of "won't serve" or "doesn't work" would be extremely helpful so that I know if you are referring to a previous issue or a new issue

Ok, well please see the above for hopefully a clear issue I'm experiencing with the new edge proxy.

will try to reproduce

I'm seeing services in another project with the edge proxy on that still have their custom domain working, so the issue may be flaky

I just sanity-check flipped the edge proxy off, confirmed that the custom domain now works, then flipped it back on, and confirmed it broke again after ~a minute.

Is there not an easy way to debug with the request ID? I thought that was one of the benefits of the new edge proxy 🤔

we would need to get a team member involved, and i will, i would just like to repro first

cc @Mig can't reproduce but likely worth looking into

hey all, thanks very much for the help in improving the new proxy. I'll see what I can find given the request IDs.

as of right now I could curl both domains and I get a 200. I know the new proxy is used because the server: railway-edge header is present for the up.railway.app domain. Cloudflare removes this.

the logs for the request id just confirm what we see in the HTML response. The domain given does not have a application. It is very weird for the up domain to work but the custom domain behind cloudflare would not.

Both domains are working right now. This may be related to something happening after a deploy because maybe you re-deployed and that fixed it ? or did just toggling the proxy fix it ?

Can you @ me if when this comes up again and I'll look at it right away. If I can see the app in this 404 state I can query the system more to see which part thinks there is no app.

the same systems power the up and custom domain routes so it's weird that one would work and not the other unless Cloudflare is sending unexpected information.

toggling the proxy should only a few minutes (the DNS ttl is 1 minute).

I can confirm that it's working for me now too. I've also enabled it for my prod service with success 👍

Not sure what fixed it between playing with the settings, redeploying, and "just waiting longer", though as you can see above it was definitely broken a bunch earlier. Will definitely hard-ping you in this thread if I see any issues crop up again.

Really appreciate the attention and support from both of you, big thanks! 😁

The redeploy fixed it for sure. There are some really rare (1%) bugs that happen after a deploy. Also, really old (before August 2023) deploys need to be backfilled in our network router. I’m doing this now.