Update — solved. It was NOT an egress/IP block; Railway's network is fine.

Following up on my post above (where I suspected Google was blocking Railway's shared US-East egress IPs) — that diagnosis was wrong, and I want to correct the record in case it saves anyone else hitting Premature close on googleapis.com.

What it actually was: an HTTP-client bug, not the network. firebase-admin mints its OAuth access token via node-fetch (through google-auth-library), and from my Railway container that call fails with ERR_STREAM_PREMATURE_CLOSE on ~100% of attempts. But Node's global fetch (undici) hits the exact same endpoint — same container, same egress IP, same moment — and succeeds.

How I proved it: I deployed a tiny /debug/egress endpoint that, from inside the container, reports the outbound IP and POSTs to the token endpoint:

egress IP: 162.220.x.x  (Railway shared)
global fetch -> https://oauth2.googleapis.com/token : HTTP 400   ✅ reachable

A 400 means we reached Google fine (it just rejected the empty test body). So the egress was never blocked. Every "fix" I tried — static IPs, IPv4-first DNS, region change (US-East → US-West), even a 2-day code rollback — failed, because they were all aimed at a network problem that didn't exist.

The fix: mint the token with global fetch instead of node-fetch. For firebase-admin you can't just swap in a custom credential — Cloud Firestore requires a real cert() credential (instanceof ServiceAccountCredential, it reads .privateKey/.clientEmail off it), so I kept cert() and overrode only its getAccessToken():

import crypto from 'node:crypto';
import { cert } from 'firebase-admin/app';

function credentialWithGlobalFetch(serviceAccount) {
  const credential = cert(serviceAccount); // keep cert() so Firestore/Storage still work
  const { client_email, private_key, token_uri = 'https://oauth2.googleapis.com/token' } = serviceAccount;
  const b64 = (s) => Buffer.from(s).toString('base64url');

  credential.getAccessToken = async () => {
    const now = Math.floor(Date.now() / 1000);
    const scope = [
      'https://www.googleapis.com/auth/cloud-platform',
      'https://www.googleapis.com/auth/firebase',
      'https://www.googleapis.com/auth/identitytoolkit',
      'https://www.googleapis.com/auth/datastore',
      'https://www.googleapis.com/auth/devstorage.read_write',
      'https://www.googleapis.com/auth/userinfo.email',
    ].join(' ');
    const head = b64(JSON.stringify({ alg: 'RS256', typ: 'JWT' }));
    const body = b64(JSON.stringify({ iss: client_email, scope, aud: token_uri, iat: now, exp: now + 3600 }));
    const sig  = crypto.sign('RSA-SHA256', Buffer.from(`${head}.${body}`), private_key).toString('base64url');

    const res = await fetch(token_uri, {              // <-- global fetch (undici), NOT node-fetch
      method: 'POST',
      headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
      body: new URLSearchParams({
        grant_type: 'urn:ietf:params:oauth:grant-type:jwt-bearer',
        assertion: `${head}.${body}.${sig}`,
      }),
    });
    if (!res.ok) throw new Error(`token mint failed: ${res.status} ${await res.text()}`);
    const j = await res.json();
    return { access_token: j.access_token, expires_in: j.expires_in };
  };
  return credential;
}

// initializeApp({ credential: credentialWithGlobalFetch(serviceAccount), ... })

Signup / login / FCM push all came back instantly.

This is the same family as the other recent Premature close thread (openai SDK / undici failing, axios working): one HTTP client chokes on a given endpoint while another handles it fine. If you see Premature close on any googleapis.com / openai.com call, try a different HTTP client before assuming it's the network.

Best guess on why it started suddenly: node:20-bookworm-slim is a moving tag, so any rebuild picks up the latest Node 20.x — a patch that tweaked HTTP/1.1 connection handling would break node-fetch (which rides Node's http module) while leaving undici untouched. Would love a pointer if anyone knows the exact Node/undici/node-fetch interaction.

Thanks to everyone who pushed back on the network theory — that's what got me to actually probe it instead of chasing IPs.