16 days ago
Project: CAPI survey
Service: backend (capi-survey.up.railway.app)
Service ID: bcdf2801-284e-463c-a449-299b01fd9778
Issue:
Every Firestore read from our backend fails on Railway with:
google.api_core.exceptions.InvalidArgument: 400 Invalid database id %28default%29
This happens on every single call to Firestore (via firebase-admin), e.g. db.collection("surveys").document(id).get(). It has taken our entire app down — no data can be read or written.
What we've ruled out:
Not a Firestore/project config issue — confirmed via firebase firestore:databases:list that the database is correctly projects/capi-hub/databases/(default) (STANDARD edition, FIRESTORE_NATIVE).
Not the service account key/credentials — ran our exact backend code locally using railway run (which injects the exact same production env vars, including FIREBASE_KEY_JSON) and it worked perfectly. Identical code + identical credentials + identical env vars succeed everywhere except inside a Railway-built/deployed container.
Not a dependency version issue — reproduced with multiple pinned combinations of firebase-admin (7.4.0, 7.5.0), google-cloud-firestore (2.27.0, 2.29.0), and grpcio (1.80.0, 1.83.0).
Not the Python version — reproduced on both Python 3.12 and 3.13.
Not the deployment region — reproduced identically in both southeast-asia and us-west.
Not a Google Cloud org policy / VPC Service Controls issue — confirmed no active organization policies or VPC service perimeters restrict this project at all.
What's left:
The only remaining variable is "running inside a Railway container" itself. Since it's a genuine 400 INVALID_ARGUMENT from Google's Firestore servers (not a client-side exception), something about outbound requests from Railway containers to Firestore's gRPC endpoint is getting malformed in transit. Could you check for any known networking/egress issue affecting outbound gRPC calls to firestore.googleapis.com from our containers?
1 Replies
16 days ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 16 days ago
16 days ago
This now looks very unlikely to be a Railway networking/egress problem. I found an upstream Google Python client regression that matches your error exactly.
google-api-core==2.35.0, released on August 24, 2026, has an open bug where Firestore’s default database ID (default) gets encoded as %28default%29. Google then receives that encoded value and rejects it with exactly:
400 Invalid database id %28default%29
The issue author bisected it specifically:
| google-api-core | google-cloud-firestore | Result |
|---|---|---|
| 2.30.3 | 2.27.0 | ✅ Works |
| 2.34.0 | 2.29.0 | ✅ Works |
| 2.35.0 | 2.28.1 | ❌ %28default%29 |
| 2.35.0 | 2.29.0 | ❌ %28default%29 |
So pinning firebase-admin, google-cloud-firestore, grpcio, Python, etc. would not fix this if Railway's fresh build resolved google-api-core to 2.35.0.
What I would do immediately
Add this explicitly to your requirements.txt:
google-api-core==2.34.0or at minimum:
google-api-core<2.35.0Then force a completely fresh Railway deployment.
I would prefer the exact pin temporarily:
firebase-admin==7.5.0
google-cloud-firestore==2.29.0
google-api-core==2.34.0You don't necessarily need to pin grpcio for this particular issue.
Also add this temporarily during deployment/startup:
import google.api_core
import google.cloud.firestore
import grpc
import firebase_admin
print("google-api-core:", google.api_core.__version__)
print("google-cloud-firestore:", google.cloud.firestore.__version__)
print("grpcio:", grpc.__version__)
print("firebase-admin:", firebase_admin.__version__)I'm expecting your broken Railway deployment to show:
google-api-core: 2.35.0Why railway run could work while deployment fails
This actually fits your observations very well.
Your local railway run command injects Railway's environment variables, but it does not reproduce Railway's newly built Python environment. Your local machine may already have an older dependency resolution, whereas a fresh Railway build may have run:
pip install -r requirements.txtand picked up the newly released google-api-core 2.35.0.
The upstream report specifically says that workloads without an upper bound on google-api-core can suddenly lose Firestore access on the next rebuild/redeploy after 2.35.0 was released.
There's also a second Firestore report from today showing the same %28default%29 behavior in Google Cloud compute environments, so Railway is definitely not the only environment where this symptom is occurring.
What actually broke
The suspected regression is a change in google-api-core related to REST/path validation. Parentheses aren't part of the RFC 3986 unreserved character set, so (default) is being converted to:
%28default%29But Firestore expects the database resource to effectively contain:
projects/capi-hub/databases/(default)not:
projects/capi-hub/databases/%28default%29The bug occurs even though you're using gRPC, apparently because this resource path also participates in gRPC routing metadata.
So I would not spend more time investigating Railway regions, VPCs, service accounts, DNS, gRPC egress, or Firestore configuration yet.
First deploy with:
google-api-core==2.34.0If that fixes it—as the upstream reproduction strongly suggests—then you've found the root cause.
The Google issue is here: Google Cloud Python Firestore regression issue
Because this is an active upstream regression, I can also keep an eye on the Google issue and let you know when they release a fixed version.