a month ago
I'm migrating my API between two projects in my workspace and custom domains refuse to route to the destination service, while re-attaching them to the original service binds within about 60 seconds every time.
Destination (broken): project ample-integrity (88ea1f6a-dbdd-4441-8a16-713499a02ca5), service crewclock-server (c341d301-5976-4c64-9469-68e569af5c35), environment production (aedfa41a-44a1-44bf-ae98-c52c67d5d070).
Origin (works): project keen-acceptance (4fe991eb-23e1-4281-968a-5e4d7cfeb3a4), service crewclock-server (cc4571b4-59ad-451b-ba57-43345968c9ca).
Domains: api.gearboxhospo.co.nz and api.burgermilitia.co.nz (external Cloudflare DNS, DNS-only CNAMEs, updated correctly on every change).
When attached to the destination service (customDomainCreate with projectId/serviceId/environmentId/domain/targetPort 8080):
- certificateStatus reaches CERTIFICATE_STATUS_TYPE_VALID
- dnsRecords show DNS_RECORD_STATUS_PROPAGATED, currentValue matches requiredValue
- but the edge returns 404 {"message":"Application not found"} indefinitely (40+ minutes observed, multiple edge IPs, verified with --resolve pinning)
- delete + recreate does not help; redeploying the service after attaching does not help
- the same service serves 200 fine on its generated domain crewclock-server-production-aeda.up.railway.app
Two more data points:
- Creating a third (test) custom domain on the destination service returns "Not Authorized" (trace 3224731620473714293). Is there a custom-domain quota, and could it be related to the non-binding?
- During the attempts one domain intermittently served a wrong certificate from some edge backends (SEC_E_WRONG_PRINCIPAL client-side) until I deleted and freshly re-registered it.
This caused a production outage during the cutover attempt and consumed most of the Let's Encrypt per-hostname weekly budget, so I'd like the root cause before retrying. Happy to provide more trace IDs / timestamps.
Pinned Solution
a month ago
- A new token will be generated when you attach the domain to a new/different service.
- The TXT content can be obtained through the
verificationTokenproperty understatus.
7 Replies
a month ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 1 month ago
a month ago
I'm able to access your domains just fine. Try using a different device or an incognito browser.
Attachments
a month ago
Thanks for checking — but the domains working right now is expected. They currently point at our OLD service, which we rolled back to after the migration failed. The problem is with the NEW service.
To be specific:
-
We attached api.burgermilitia.co.nz and api.gearboxhospo.co.nz to a new service (service id c341d301, project ample-integrity, id 88ea1f6a) in the same workspace (burgersforlife2026). Both showed certificateStatus VALID and DNS PROPAGATED, but Railway's edge returned 404 "Application not found" indefinitely — over an hour, including after fresh redeploys. The service itself was healthy the whole time (its railway.app domain responded fine).
-
Re-attaching the exact same domains back to the old service (service id cc4571b4, project keen-acceptance) worked within ~60-90 seconds. That's the state you tested — it works because it's the rollback, not because the issue is resolved.
-
Separately, attempting to create a third custom domain returned "Not Authorized" — trace id 3224731620473714293. Is there a per-account or per-service custom domain quota we're hitting?
So the questions remain: why do custom domains never route when attached to the service in ample-integrity, and what does the "Not Authorized" mean? We can't just keep retrying to reproduce it for you — the failed attempts burned through Let's Encrypt's 5-certs-per-week limit on api.burgermilitia.co.nz, so we're frozen on domain changes until the window resets (~26 July). We'd like to understand the cause before attempting the cutover again.
0x5b62656e5d
Also, did you update the TXT record when adding the domains?
a month ago
That's very likely it — thank you. Checking our Cloudflare zones now, both domains have _railway-verify TXT records (e.g. _railway-verify.api.burgermilitia.co.nz) that were created when the domains were first set up in May/June and never modified since. When we moved the domains to the new service we only updated the CNAME targets, not those TXT records.
Two questions to confirm before we retry:
-
Is a new railway-verify token generated each time a domain is attached (or per service/project)? I.e. was the stale TXT token the reason the edge kept returning 404 "Application not found" on the new service even though the certificate showed VALID and DNS showed PROPAGATED?
-
We attached the domains via the GraphQL API — querying domains { customDomains { status { dnsRecords } } } for our currently-attached (working) domains only returns the CNAME requirement, no TXT. Does the TXT requirement only appear in the response at attach time / while unverified? Just want to make sure we capture the right value on the next attempt.
And thanks for the Hobby plan clarification on the 2-domains-per-service limit — that explains the "Not Authorized" error.
We'll retry the migration with fresh TXT values once Let's Encrypt's per-hostname weekly cert limit resets for us (~26 July — the failed attempts used up the 5/week budget on one of the domains).
0x5b62656e5d
Also, did you update the TXT record when adding the domains?
a month ago
That's very likely it — thank you. Checking our Cloudflare zones now, both domains have _railway-verify TXT records (e.g. _railway-verify.api.burgermilitia.co.nz) that were created when the domains were first set up in May/June and never modified since. When we moved the domains to the new service we only updated the CNAME targets, not those TXT records.
burgersforlife2026
That's very likely it — thank you. Checking our Cloudflare zones now, both domains have _railway-verify TXT records (e.g. _railway-verify.api.burgermilitia.co.nz) that were created when the domains were first set up in May/June and never modified since. When we moved the domains to the new service we only updated the CNAME targets, not those TXT records. Two questions to confirm before we retry: 1. Is a new railway-verify token generated each time a domain is attached (or per service/project)? I.e. was the stale TXT token the reason the edge kept returning 404 "Application not found" on the new service even though the certificate showed VALID and DNS showed PROPAGATED? 2. We attached the domains via the GraphQL API — querying domains { customDomains { status { dnsRecords } } } for our currently-attached (working) domains only returns the CNAME requirement, no TXT. Does the TXT requirement only appear in the response at attach time / while unverified? Just want to make sure we capture the right value on the next attempt. And thanks for the Hobby plan clarification on the 2-domains-per-service limit — that explains the "Not Authorized" error. We'll retry the migration with fresh TXT values once Let's Encrypt's per-hostname weekly cert limit resets for us (~26 July — the failed attempts used up the 5/week budget on one of the domains).
a month ago
- A new token will be generated when you attach the domain to a new/different service.
- The TXT content can be obtained through the
verificationTokenproperty understatus.
Status changed to Solved 0x5b62656e5d • about 1 month ago