Custom domains return x-railway-fallback 404 despite VALID cert — edge not routing to consumer service
axiomstate
PROOP

a month ago

Project: qisstpay (id 71932004-50be-46ce-be77-8631393f96bf), environment production. Service: consumer (Next.js).

Two custom domains on the consumer service return HTTP 404 with header "x-railway-fallback: true" on every path, even though the certificate shows VALID and DNS is fully propagated:

  • app.qisstpay.com (CNAME -> ix0492fy.up.railway.app, PROPAGATED, cert VALID)
    • docs.qisstpay.com (CNAME -> q89pk9sz.up.railway.app, PROPAGATED, cert VALID)

The root domain qisstpay.com on the SAME consumer service works perfectly (200), and other services' custom domains (e.g. api.qisstpay.com on the api service) also work. The default *.up.railway.app URL for consumer serves fine too. So the app is healthy — the edge simply is not binding these two hostnames to the consumer service.

Example (404 fallback, not the app):

curl -sI https://app.qisstpay.com/ -> HTTP/2 404, server: railway-hikari, x-railway-fallback: true, x-railway-edge: lax1

Already tried, no effect: redeployed the consumer service; deleted + re-added the custom domain (updating DNS to the new CNAME target each time); triggered certificate issuance (cert went VALID, routing still 404s). Control plane reports the domain VALID with correct/propagated DNS, but the edge returns the fallback 404 — looks like a stuck edge binding for the consumer service's non-root custom domains.

Could you please re-bind app.qisstpay.com and docs.qisstpay.com to the consumer service on the edge? Happy to provide any additional IDs. Thanks!

Solved$20 Bounty

Pinned Solution

Your TXT record on _railway-verify.app.qisstpay.com is malformed. There's two railway-verify=. (Though I'm somehow able to access app.qisstpay.com just fine)

7 Replies

Railway
BOT

a month ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway 30 days ago


axiomstate
PROOP

a month ago

Follow-up: verification done to pre-empt the usual user-side suspects, so you can go straight to the edge state.

  1. Duplicate registrations: audited every project x environment x service in our account via the API — each qisstpay.com hostname is registered exactly once (9 domains, all in project qisstpay / production). No duplicates anywhere.

  2. DNS/registrar: zone is served solely by Cloudflare (fay + rodney NS at the .com registry), DNSSEC unsigned (no stale DS from the Google Domains -> Squarespace migration), and Google/Cloudflare/Quad9 resolvers all return identical answers. No wildcard records.

  3. Clean reproducer — docs.qisstpay.com (DNS untouched, NOT proxied, grey-cloud CNAME straight to q89pk9sz.up.railway.app): at the same moment, openssl shows your edge serving a valid cert CN=docs.qisstpay.com (issued Jul 18) while HTTP returns 404 with x-railway-fallback: true (x-railway-edge: lax1). Control: merchant.qisstpay.com — identical DNS-only CNAME pattern, same zone, different service — serves its cert AND routes 200.

  4. So the edge holds the certificate for the hostname (SNI table knows it) but routes it to fallback (routing table doesn't) — an internally inconsistent state we can't produce from our side. Possibly related: control plane shows the working domains on edgeId edge-6705e5cc... and both broken ones on edge-500b32d4..., and neither broken domain obtained a cert until we manually called customDomainIssueCertificate — they appear stuck since creation.

Note: app.qisstpay.com currently 301s to the root via Cloudflare as a user-facing workaround, so please test against docs.qisstpay.com — it still exhibits the raw symptom.


Your TXT record on _railway-verify.app.qisstpay.com is malformed. There's two railway-verify=. (Though I'm somehow able to access app.qisstpay.com just fine)


Also, it may take up to a few hours (maybe even a day) for Railway to validate the DNS and issue certificates. If one isn't issued by then, try removing the domain from Railway and add it back after ~10-15 mins. Update DNS records as necessary.


axiomstate
PROOP

a month ago

Good catch, thank you! Fixed and in progress:

  1. TXT on _railway-verify.app corrected to a single railway-verify= prefix (propagated on 1.1.1.1 and 8.8.8.8). Worth noting where the doubled prefix came from: the railway domain CLI prints the Value column as "railway-verify=railway-verify=" — it got pasted faithfully at setup time. Might be worth a CLI bug report.

  2. Heads up: you're able to access app.qisstpay.com because we put a Cloudflare 301 -> qisstpay.com in front of it as a user-facing workaround (it's proxied now, which is also why its CNAME check shows REQUIRES_UPDATE). The raw symptom is still reproducible on docs.qisstpay.com — grey-cloud CNAME straight to your edge, valid cert served, 404 x-railway-fallback.

  3. Interesting datapoint: docs' TXT was already well-formed (single prefix), and docs shows the same routing failure — so the malformed TXT may not be the whole story.

  4. Following your advice, I've removed docs.qisstpay.com and will re-add after ~15 minutes, then update DNS to the new target and report back here once it's had time to validate.


axiomstate

Good catch, thank you! Fixed and in progress: 1. TXT on _railway-verify.app corrected to a single railway-verify= prefix (propagated on 1.1.1.1 and 8.8.8.8). Worth noting where the doubled prefix came from: the `railway domain` CLI prints the Value column as "railway-verify=railway-verify=<token>" — it got pasted faithfully at setup time. Might be worth a CLI bug report. 2. Heads up: you're able to access app.qisstpay.com because we put a Cloudflare 301 -> qisstpay.com in front of it as a user-facing workaround (it's proxied now, which is also why its CNAME check shows REQUIRES_UPDATE). The raw symptom is still reproducible on docs.qisstpay.com — grey-cloud CNAME straight to your edge, valid cert served, 404 x-railway-fallback. 3. Interesting datapoint: docs' TXT was already well-formed (single prefix), and docs shows the same routing failure — so the malformed TXT may not be the whole story. 4. Following your advice, I've removed docs.qisstpay.com and will re-add after ~15 minutes, then update DNS to the new target and report back here once it's had time to validate.

For 3, the docs domain seemed to be configured just fine. Probably just needed a longer wait or a remove/readd process.


axiomstate
PROOP

a month ago

Confirmed fixed for docs — the remove/re-add cycle worked. Removed the domain, waited ~12 minutes, re-added, pointed the CNAME at the new target (ghza56mb), and docs.qisstpay.com now serves 200 from the app with no x-railway-fallback, on the same lax1 edge that was 404ing. Running the identical cycle for app.qisstpay.com now; once it binds I'll drop the Cloudflare redirect workaround and report back.

One small thing for the team: the railway domain CLI reproducibly prints the TXT Value column as railway-verify=railway-verify= (doubled prefix) — it did it again on this re-add. Easy for users to paste verbatim and end up with a malformed record.


axiomstate
PROOP

a month ago

All resolved — app.qisstpay.com is also fixed via the same cycle (remove → wait ~12 min → re-add → point CNAME at the new target, grey-cloud). Both domains now serve 200 directly from the edge with their own valid certs, no fallback:

  • app.qisstpay.com → 200 (CN=app.qisstpay.com)
    • docs.qisstpay.com → 200 (CN=docs.qisstpay.com)

Cloudflare redirect workaround retired (record un-proxied, so the rule no longer applies). Accepted your remove/re-add answer as the solution — thanks for the fast help. Only open item for the team is the CLI's doubled railway-verify= prefix in the TXT output, which reproduced on all three re-adds today.


Status changed to Solved axiomstate 30 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...