High rate of 502s in production environment

13 days ago

I'm currently (for the last ~3 hours) observing a high 502 error rathe in my API service. This is happening during the build of my Next.js project that consumes the API via its public URL. The large amount of concurrent requests issued by Next.js reliably triggers the 502 in every build. But this behavior is not reproducible elsewhere: not in my local development environment, and also not in the Railway development environment which carries almost identical data. There are also no error logs in my API service, and so far it could handle the build-time-load easily.

So I'm starting to wonder whether something outside my control is causing issues here?

Solved$20 Bounty

12 Replies

Railway
BOT

13 days ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway 13 days ago


Can you share more information about this? Like exact error logs, if you're using a custom domain and proxying your requests (e.g., Cloudflare proxy). If Next.js is failing to reach the API during the build phase, does it also fail when deployed?


13 days ago

It looks like the API server was overwhelmed after all. Limiting the concurrent static generation in Next.js resolved the error for now. Not sure why I couldn't reproduce the issue in the dev environment.


Check connection pooling for the database to handle sudden spikes and requests for API traffic.

Review rate limiting rules for API traffic.


taig

It looks like the API server was overwhelmed after all. Limiting the concurrent static generation in Next.js resolved the error for now. Not sure why I couldn't reproduce the issue in the dev environment.

Also configure railway private networking so that your next.js build can call your API using an internal url instead of using the public one


h701h

Also configure railway private networking so that your next.js build can call your API using an internal url instead of using the public one

12 days ago

I don't think that's possible, is it? The railway build environment and the project environment are not on the same network, at least that's how I remember it?


taig

I don't think that's possible, is it? The railway build environment and the project environment are not on the same network, at least that's how I remember it?

As your project environment is isolated from the build environment, try bypassing the public network entirely and avoiding 502 errors.

You should be configuring private networking in railway, Your next.js build can connect to API service using the internal connection string provided by railway's private networking which lets the build bypass the public url completely.

You can also use build time environment variables to point your next.js app to the public URL specifically for the build process while keeping the internal URL for runtime communication only.


taig

I don't think that's possible, is it? The railway build environment and the project environment are not on the same network, at least that's how I remember it?

Tell me if you need steps to configure private networking in railway


h701h

As your project environment is isolated from the build environment, try bypassing the public network entirely and avoiding 502 errors. You should be configuring private networking in railway, Your next.js build can connect to API service using the internal connection string provided by railway's private networking which lets the build bypass the public url completely. You can also use build time environment variables to point your next.js app to the public URL specifically for the build process while keeping the internal URL for runtime communication only.

12 days ago

I am using private networking for my deployed next.js app, but the public URL during the railpack build. I'd love to use internal networking for the build as well, but I do not think this is possible, because as you say: "your project environment is isolated from the build environment"


taig

I am using private networking for my deployed next.js app, but the public URL during the railpack build. I'd love to use internal networking for the build as well, but I do not think this is possible, because as you say: "your project environment is isolated from the build environment"

  1. Go to railway dashboard and select your next.js service.
  2. Go to variables tab. Add a build time environment variable that points your public API url for the build process.
  3. Add another environment variable for runtime using the internal private network address using the format 'service-name.railway.internal'

Once these variables are set, your next.js App would build successfully and can communicate with other services using the internal private network.


romonaga
PRO

12 days ago

Where possible you should use the internal DNS to reach your application from inside of Railway,this is also a cost saver as well as performance. The only time you should use the External is when well, you actually want people to hit that endpoint. Also, if you are doing lots of work, check how many CPUI's and memory you have allocated.


taig

I am using private networking for my deployed next.js app, but the public URL during the railpack build. I'd love to use internal networking for the build as well, but I do not think this is possible, because as you say: "your project environment is isolated from the build environment"

What's your current status?


h701h

1. Go to railway dashboard and select your next.js service. 2. Go to variables tab. Add a build time environment variable that points your public API url for the build process. 3. Add another environment variable for runtime using the internal private network address using the format 'service-name.railway.internal' Once these variables are set, your next.js App would build successfully and can communicate with other services using the internal private network.

Kindly mark the thread as solved


Status changed to Solved 0x5b62656e5d 9 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...