PayPal API returning 403 Forbidden (Varnish Error 54113) only from Railway

ciocca-dev
PROOP

17 days ago

We're experiencing a 403 Forbidden error with Varnish Error 54113 when making requests to PayPal's OAuth token endpoint (https://api-m.paypal.com/v1/oauth2/token) from our Vendure application hosted on Railway.

Key Details:

  • Same credentials work perfectly in Postman, curl, and other environments outside Railway

  • Error occurs consistently only when requests originate from Railway infrastructure

  • PayPal's Varnish cache server is blocking our requests with error code 54113

  • We've already implemented:

    • app.set('trust proxy', true) in Express configuration

    • Proper Basic Authentication with Base64 encoded credentials

    • Clean credentials (no whitespace)

Error Response:

403 Forbidden
Error 54113
Details: cache-ams21077-AMS
Varnish cache server

Is there any Railway-specific networking configuration (IP reputation, headers, IPv6/IPv4 routing) that could cause PayPal's WAF to block our requests? The fact that identical requests work from other networks suggests a Railway infrastructure-related issue.

Please advise if there are any known compatibility issues with PayPal's API or if Railway IPs might be flagged/blocked.

$20 Bounty

9 Replies

Railway
BOT

17 days ago


17 days ago

This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.

Status changed to Open brody 17 days ago


luigigorlero
PRO

16 days ago

The issue seems to happen with Railway’s EU server. Enabling Static IP does not fix the issue, moving our frontend to a different region (like US East) fixes it !


caullenomdahl
HOBBY

15 days ago

PayPal blocks certain Railway EU datacenter IPs. cache-ams21077 means you're hitting their Amsterdam cache server - that's where the block happens.

Based on what luigi found, switching to us-west1 or us-east4 fixes it. Static IP won't help since Railway's IPs can be shared, and if someone else on that IP triggered PayPal's filters, you're blocked too.

If you need to stay in EU:

  • Route PayPal calls through a proxy service

  • - Or spin up a small VPS elsewhere ($6/mo DO droplet) just for PayPal OAuth requests

You can test if it's definitely IP-based by curling the endpoint from your Railway container vs locally - if local works and Railway doesn't, confirmed.

Region change is fastest fix though.


ciocca-dev
PROOP

15 days ago

At the moment, we have resolved the error by hosting the PayPal confirmation on another server as a provisional measure. Railway MUST investigate this major issue IMMEDIATELY. This can cause significant economic losses, so this MUST be resolved


ciocca-dev

At the moment, we have resolved the error by hosting the PayPal confirmation on another server as a provisional measure. Railway MUST investigate this major issue IMMEDIATELY. This can cause significant economic losses, so this MUST be resolved

Unsure as to why you're pinning this on Railway. PayPal is returning 403, meaning the error is reaching PayPal's server. You should contact them regarding this issue.

Edit: What @caullenomdahl is correct. This is not Railway's responsibility, PayPal has for some reason blocked Railway.


ciocca-dev
PROOP

12 days ago

How did you think this was our responsibility? Every other service is working fine. If PayPal has pined Railway's IP addresses as spam, there must be something wrong with Railway's infrastructure. We cannot contact PayPal on Railway behalf because this is a Railway infrastructure problem. As @caullenomdahl mentioned above, they also had to move to another server to resolve this issue


samgordon

Unsure as to why you're pinning this on Railway. PayPal is returning 403, meaning the error is reaching PayPal's server. You should contact them regarding this issue.Edit: What @caullenomdahl is correct. This is not Railway's responsibility, PayPal has for some reason blocked Railway.

luigigorlero
PRO

9 days ago

Since Railway is providing the servers, it should also make sure its IPs don’t end up blacklisted anywhere, either by enforcing the correct use of services on the platform (the agreement we have to check during onboarding simply won’t do) or having regular checks with service providers to ban misbehaving users


luigigorlero

Since Railway is providing the servers, it should also make sure its IPs don’t end up blacklisted anywhere, either by enforcing the correct use of services on the platform (the agreement we have to check during onboarding simply won’t do) or having regular checks with service providers to ban misbehaving users

Railway has already likely removed the user causing the havoc, however it's not their responsibility to then go out and attempt to get the IP unblacklisted. There are thousands of potential services and Railway isn't monitoring them.

There is not something "wrong with Railway's infrastructure".


samgordon

Railway has already likely removed the user causing the havoc, however it's not their responsibility to then go out and attempt to get the IP unblacklisted. There are thousands of potential services and Railway isn't monitoring them.There is not something "wrong with Railway's infrastructure".

ciocca-dev
PROOP

9 days ago

I respectfully disagree that this absolves Railway of responsibility in this situation.

Why This IS Railway's Responsibility

We cannot contact PayPal on Railway's behalf. PayPal won't unblock IPs based on requests from individual customers - they need to hear from the infrastructure provider directly. Railway has the business relationship and credibility to address this with PayPal, we don't.

Shared infrastructure = shared responsibility. When Railway provides shared IP addresses, maintaining the reputation of those IPs becomes part of the service we're paying for. If another Railway customer abused the service and got the IPs blacklisted, that's an internal Railway issue that shouldn't impact paying customers.

This is causing severe financial damage to our business. We're losing real revenue because Railway's EU infrastructure is blocked by a major payment provider. This isn't a minor inconvenience - it's actively harming our operations.

Why Suggested Solutions Don't Work for Us

  • Changing regions: Not feasible - our business operates in Europe, and moving to US regions would significantly impact latency and user experience

  • Dedicated IPs: While this might work, we shouldn't have to pay additional fees to work around Railway's IP reputation problems

  • External proxy: This adds unnecessary complexity, cost, and another point of failure to our production infrastructure

Our Experience

We have never received this kind of treatment from any hosting provider. Every other service we've used either proactively monitors IP reputation or takes responsibility for resolving blacklist issues when they occur.

Railway needs to either:

  1. Contact PayPal directly to get your EU datacenter IPs unblocked

  2. Implement better abuse prevention to keep your IPs off blacklists

  3. Provide working dedicated IPs at no additional cost as compensation for this service failure

This is a critical infrastructure issue affecting multiple paying customers (has seen above). It requires Railway's direct intervention, not workarounds that shift the burden to users.


Loading...