3 hours ago
Project truthful-caring (ee1d7a37-1af2-491d-9fcf-0bbfe52b76ed), production environment (58ec7745-b8b6-45cd-8bdd-eb64dbe39a19), service Supa (dcb68bad-37c7-4e88-8e29-173611046334).
MobileMessage webhook delivery stopped after 2026-09-19 14:40:18 UTC. Last known successful source IP: 209.38.21.135 (current egress confirmation pending). Their delivery worker reports connection timeout/no response for both supa-production-060b.up.railway.app and ai-jane.com.au, path /api/v1/sms-webhook/mobilemessage. The earlier investigation found no deployment at onset and no subsequent provider HTTP requests across the deployment history.
Fresh evidence on deployment 45896aab-2035-4a18-a211-74a0258cf288: during 2026-09-21 07:20 to approximately 07:50 UTC, 179 HTTP requests were logged; 17 webhook requests were all our local probes from 115.70.99.7. Successful probes to both hostnames used HTTP/2, returned 200 in 3-12 ms, and were handled by edge region asia-southeast1-eqsg3a. Example successful request ID: R1_bSBCKQFeJhlXfmrpb1w at 07:39:26.765263940 UTC on ai-jane.com.au.
Please investigate edge filtering/rate limits and TCP/TLS routing for the provider source IP, including any changes around onset. Can your edge telemetry show connection attempts that never produce HTTP logs? MobileMessage says it cannot expose its worker diagnostics to frontline support, so we are requesting engineering escalation there too.
Bypass assessment
A Cloudflare relay could test a different ingress route, but the handoff's proposed immediate-200/background-forward design has a failure gap: the provider stops retrying as soon as it sees success, even if forwarding subsequently fails. If built, acknowledge only after durable queue acceptance, then forward with bounded retries and dead-letter handling. Keep retries within the backend's 12-hour dedup retention; manually review expired events rather than blindly replaying days later. Keep the existing poller.
Changing MOBILEMESSAGE_INBOUND_MIN_AGE_SECONDS to 30 would reduce the nominal grace-plus-poll delay to roughly 30-90 seconds, plus processing/scheduling time. It does not repair webhook delivery. This is an undeployed option requiring the explicit production-change approval specified in AGENTS.md and docs/INSTRUCTION_AUTHORITY.md section 4
1 Replies
3 hours ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 3 hours ago
2 hours ago
ive independent Globalping probes on DigitalOcean AS14061 in Sydney all resolve both application hostnames and establish TCP connections, but time out during the TLS handshake after 15 seconds. The same five probes also fail TLS to railway.com and successfully reach https://www.cloudflare.com. This reproduces a failure outside the application and outside MobileMessage's own workers.
DigitalOcean's published geofeed maps 209.38.16.0/20 to Sydney, AU-NSW. MobileMessage's historical source IP 209.38.21.135 is inside this prefix. RIPEstat identifies the announced prefix's network as AS14061 (DigitalOcean).
Interpretation: a regional DigitalOcean Sydney-to-Railway TLS path problem is now strongly supported. It is a plausible explanation for MobileMessage's outage. It remains unconfirmed whether their current worker uses the historical IP/path. These tests do not establish which operator is responsible, whether the issue began at the recorded outage onset, or whether the specific mechanism is edge filtering, TLS termination, path MTU/packet loss, or another transport defect. A traceroute through Tokyo does not prove the location of the faulty component.
Results
| Source and target | Result |
|---|---|
| 5 DigitalOcean Sydney probes -> ai-jane.com.au HTTPS /health | 5/5 TLS handshake timeouts, ~15,000 ms; DNS 3-9 ms, TCP ~107 ms |
| 5 DigitalOcean Sydney probes -> supa-production-060b.up.railway.app HTTPS /health | 5/5 TLS handshake timeouts, ~15,000 ms; DNS 135-151 ms, TCP 107-108 ms |
| Same 5 Sydney probes -> railway.com HTTPS / | 5/5 TLS handshake timeouts |
| Same 5 Sydney probes -> www.cloudflare.com HTTPS / | 5/5 HTTP 200, 244-255 ms; TLS 6-12 ms |
| 2 DigitalOcean Singapore probes per application hostname | 4/4 HTTP 200; TLS 5-10 ms; total 22-211 ms |
| 2 DigitalOcean Sydney probes per application hostname, plain HTTP /health | 4/4 HTTP 301 HTTPS redirects, 218-366 ms |
| Initial geographically distributed HTTPS checks | 17/18 HTTP 200, 21-881 ms; only DigitalOcean Sydney failed |
| Local TLS 1.2 checks, both hostnames pinned to each of the two Railway IPs | 4/4 HTTP 200, 0.85-1.08 seconds; hostname verification retained |
| Google and Cloudflare DNS resolvers | Agree: ai-jane.com.au = 69.46.46.116; Railway hostname = 69.46.46.67; no AAAA records |
| Railway service WAF Under Attack Mode | Disabled |
| Railway domain settings | Both application hostnames ACTIVE, targeting port 8080 |
The five Sydney probe tags were u-mdspur, u-metajar:digitalocean, u-MORGiON666, u-smf13, and u-MartinKolarik. They are distinct probes but share a hosting provider and region; this does not establish failure across every Sydney network.
Route evidence
TCP port 443 traceroutes from two DigitalOcean Sydney probes to each application hostname reach the destination in approximately 106-108 ms. Visible transit goes from DigitalOcean's au-alx1 routers through NTT Sydney and Tokyo routers before reaching the Railway anycast IP. Singapore DigitalOcean control routes instead go through local Singapore transit and reach the destination in about 2-9 ms.
Plain HTTP succeeds from Sydney and returns Railway's HTTPS redirect. Therefore, DNS and basic IP/TCP connectivity work in this reproduction. TLS is the stage where progress stops. Switching the registered webhook to plain HTTP is not a fix: it redirects to the same failing HTTPS path and would expose the query secret in transit.