25 days ago
Hello Railway team,
Our existing production FastAPI service (cass-lab) is running, but previews that read PDFs from a Railway storage bucket fail. We need help diagnosing this without changing stored data or blindly redeploying.
Verified observations:
- Question lists work. Sample IB, CIE and TMUA PDF preview endpoints return HTTP 422 with our application's sanitized error: "Cloud source PDF unavailable for storage key". An AP preview that does not use this read path returns HTTP 200.
- The application catches all S3 read exceptions, so the underlying runtime error is still unknown.
- The current production deployment was created on 2026-10-21 at 01:47:23 UTC. Its last visible application startup completed at 01:50:11 UTC that day.
- Railway bucket metadata reports the current credentials were issued on 2026-10-21 at 22:37:15 UTC, after that application startup.
- Using the currently resolved service variables locally, HEAD succeeded for both tested source PDFs and one complete PDF GET succeeded. Another local body read timed out. This does not prove access from the existing container.
- DNS resolution and direct TLS certificate verification for the storage endpoint succeed.
Our working hypothesis is that the running application may retain obsolete credentials. This is NOT yet confirmed; we have not inspected its environment or the actual S3 exception.
Questions:
- After bucket credentials change, does restarting the existing deployment refresh referenced AWS variables, or does restart reuse deployment-snapshotted values? What is the smallest supported recovery action that preserves the exact current image and persistent volume, without a new source upload or rebuild?
- How can we independently verify the official current SSH host public key/fingerprint for ssh.railway.com? A user-approved temporary registered key reached "Host key verification failed" before any container command ran. That key has been revoked and its removal verified. We did not disable host-key checking or trust an unverified host. Please provide an authoritative verification method so we can perform a read-only runtime check.
- Can Railway inspect server-side object-read/authentication failures for the selected service/bucket, or confirm whether credential lifecycle changes explain this behavior?
Please advise only at this stage; do not rotate credentials, restart/redeploy, or modify resources on our behalf. We will authorize the minimal corrective action separately after the cause is verified. No secrets or source PDFs are attached. This is a separate issue from earlier source-snapshot/upload incidents.
Thank you.
5 Replies
25 days ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 25 days ago
25 days ago
For Q1, you’ll need to redeploy the service for it to use the new credentials.
For Q2, there isn’t a way. Keys aren’t publicly available.
25 days ago
Thank you for confirming that a redeploy is needed to load the new bucket credentials. Recovery is now blocked by source snapshot creation.
Failed deployment ID: bf0b8850-9004-4ca9-88f2-d30f162e0bbf
Created: 2026-10-26 18:43:08 UTC (2026-10-27 02:43:08 Asia/Shanghai).
Service: cass-lab, production.
Redeploying the existing successful version was rejected because its source was no longer available to rebuild and there was no build to copy. We then made exactly one source-upload attempt using the retained original release package. All 8,980 files matched the original upload manifest by path, size and SHA-256, with no extra files or package changes.
After approximately 131 seconds, the CLI returned:
UPLOAD_FAILED: Failed to upload code with status code 524
The resulting deployment is FAILED, with configErrors:
Failed to create code snapshot. Please review your last commit, or try again.
If this error persists, please reach out to the Railway team.
The previous successful deployment remains running. No new runtime went online, so the credential refresh and PDF recovery remain incomplete. We have stopped and will not repeat deployment attempts.
Could Railway inspect the backend upload/ingress and source-snapshot records for this deployment and identify the exact cause? Please check archive receipt/validation, snapshot storage/registration, and any timeout or rejection before the build stage. Please also advise the supported way to redeploy this exact application with current bucket credentials when the previous build cannot be copied.
Please investigate and advise only; do not deploy, restart, rotate credentials, or modify resources on our behalf. No credentials, source PDFs or raw private logs are attached. Thank you.
25 days ago
Try selecting your service, press Cmd/Ctrl + K, and select deploy latest commit (or source image).
If that doesn’t work, I’d try recreating a new service instead.
25 days ago
Thank you for the suggestion. We checked the command palette without executing any action.
With cass-lab selected in the production environment, Cmd+K and a search for "deploy" show only:
- Deployments
-
- Service Deployments
-
- Latest Deployment
There is no "Deploy latest commit", "Redeploy service image", or "Redeploy latest deployment" option, and no executable deployment target, commit, or source image is shown. We closed the palette without selecting any command.
This service was deployed via Railway CLI source uploads using our prepared release package and Dockerfile. It has no configured GitHub repository or external source image. The original successful deployment is still ACTIVE/Online; the latest failed CLI upload (bf0b8850-9004-4ca9-88f2-d30f162e0bbf) stopped at Initialization > Snapshot code, with Build and Deploy not started. Redeploying the original successful version was already rejected because its source/build could no longer be copied.
Could you please provide specific supported steps for this CLI-uploaded service with no repository/image source, given that the command-palette deployment options are absent? Can Railway restore access to the existing successful build/source, or resolve the source-snapshot creation failure for the deployment above? Please clarify which exact operation and deployment target would preserve our current application, persistent volume, domain and user data while loading the current bucket credentials.
We have not attempted another deployment and have not created a new service. Please investigate and advise only; do not deploy, restart, create services, rotate credentials, or modify resources on our behalf. Thank you.
24 days ago
I'd try creating a blank service, then running railway up into that service.
If that works, you can simply unmount the volume from the current service and mount it onto the new one.