During build the static IP is not used.

It's not during build.

It would be during build.

Our last deployment was 18 hours ago, all of the 401s we've seen are since then.

Tho I'll clarify that in my original post I typo'd June 2nd instead of June 3rd - this started today.

(I'll fix that)

Besides, all of these outgoing requests are user-triggered which can only happen from an active deployment.

That is a DataCamp IP. We use DataPacket for builds; zero user workloads run on them.

I promise you that these requests are not occuring during builds. We have 0 code paths that could ever hit this. It requires an active user session which simply could not exist at build time, unless your build agents know how to OAuth login with Google.

It is occurring during some build somewhere on the platform, again we do not run user workloads on DataPacket.

Nor do we run the egress gateways on DataPacket.

X-Forwarded-For: 162.220.234.15, 89.222.103.194

Thats inbound, this thread's title says outbound.

I mean, it's technically both I guess. We send a request from a service in one project, out over its public URL, and then back into a service in another project.

It at least appears that the outbound IP is incorrect.

I get what you are saying, but it's not both; they are very distinct.

89.222.103.194 in X-Forwarded-For is the CDN's IP.

I gave this answer with the limited information available at the time, so with this new information, this is no longer correct.

That's my bad, I didn't think to actually hit it in the terminal.

The application is likely using X-Forwarded-For, not X-Real-Ip

But I am confused why it has worked for weeks and suddenly the extra IP has come in <:thinkingbread:935682467124359258>

Also, friendly bit of insider information, since you said "only a single IP per API key."

See that "Enable HA Static IPs" button? Eventually, that will be automatically enabled for all workloads in the future and then you will need to whitelist multiple IPs. We will, of course, send out multiple communications long before this happens, so me saying this is just giving you an extra long heads-up.

So in short, modify your code to use X-Real-IP and accept multiple IPs for the allowlist.

New CDN, you didn't see? - https://discord.com/channels/713503345364697088/921233523719946260/1510042109976121424

I appreciate that, but unfortunately this is an off-the-shelf product we use for support ticket handling. I can maybe write a patch for the build stage to fix it up, but time is against me. <:madman3D:605090841102647297>

Totally missed it. 😂 It's been a mad few weeks. <:Peepo_Cry:832227339161174027>

We haven't even put a date on the hard cutover; you have plenty of time.

My favourite solution to this would be inter-project networking, but we can dream ;P

Anyways, I appreciate your help in figuring this out! Thanks so much! ❤️

Maybe the best solution is to not use IP allowlists. You don't need to be a Railway employee to know that anyone can get the same IPs you are using just by paying for Pro.

Could technically be done with Tailscale.

I'll just disable the IP check I think. Never been a fan of them anyway. The key itself is large enough that I doubt it'll ever matter.

Sounds good

And it's a write-only API anyway, so what are they gonna do... send us support requests? 😂

As a support engineer, that is about the worst thing you can have happen to you.