6 days ago
Project: adaptable-luck (119455b3-fadc-470b-b192-c8e9ccc4fb9f)
Environment: production (5c37e9c5-52dc-4557-8177-2d6d585789da)
Service: airtag-tracker (80413133-4e85-4a39-8955-731ce77edf0c)
Deployment: fd54d81c-a938-46b4-847f-04ba18304b2f
Region: sfo
Timeline:
- Last successful location fetch: 2026-08-14 08:36:54 UTC
- First all-zero poll: 2026-08-14 08:40:21 UTC
- Failure remains active now.
The service remains healthy and runs its five-minute scheduler, but outbound
TCP traffic to gateway.icloud.com is dropped by Railway networking:
-
DNS queries for gateway.icloud.com succeed.
-
Recent network logs: 113 Apple-bound TCP/443 egress flows in 20 minutes;
all have dropCause=IP_RPFILTER.
-
The application then times out while fetching from Apple for every connected
account, so no locations or snapshots are saved.
Please investigate why this deployment’s egress traffic to the resolved
gateway.icloud.com IPs (17.248.193.x) is being dropped with IP_RPFILTER,
and advise whether the instance or its network path should be replaced.
I can provide exported network-log entries if useful.
2 Replies
6 days ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 6 days ago
6 days ago
Recovery update — I restarted airtag-tracker at 2026-08-15 13:47 UTC.
The service recovered immediately
The new instance’s Apple-bound traffic is no longer showing IP_RPFILTER. Before the restart, Railway network logs recorded 148 dropped TCP/443 flows to gateway.icloud.com; post-restart flows are completing without that drop cause.
This confirms the outage was tied to the previous instance/network path. Please investigate the IP_RPFILTER drops and confirm the durable platform-side resolution.
6 days ago
Additional symptom: Google OAuth was also affected.
Between 2026-08-15 13:30:13 and 13:32:15 UTC, five consecutive Google login attempts reached our callback but then returned users to the login page.
GET /auth/google/login completed normally (303 in 3–26 ms). After the user selected their Google account, /auth/google/callback took approximately 10 seconds (10,019–10,094 ms) before returning 303 to /login.
Application log for each attempt:
oauth google login failed: token exchange failed: [Errno 101] Network is unreachable
The failing operation is the server-side POST to:
https://oauth2.googleapis.com/token
which exchanges Google’s authorization code for tokens. No user session could be created, so the app redirected back to login.
This is not consistent with a bad OAuth redirect URI, client ID, or client secret: those failures should reach Google and return an HTTP error. Here the application failed locally while opening its outbound network connection.
After restarting airtag-tracker at 2026-08-15 13:47 UTC, a direct request from the new instance to the same Google token endpoint reached Google immediately (returning the expected HTTP 400 when sent without an authorization code). Google login connectivity therefore recovered with the restart.
Please correlate this Google OAuth egress failure with the prior instance’s Apple-bound TCP/443 flows dropped with IP_RPFILTER, and investigate the old instance/network path.