17 days ago
Since ~15:00 UTC today, every deployment on my account fails at the snapshot stage with "Failed to snapshot repository" — the build never starts, deployment meta shows configErrors: ["Failed to snapshot repository. Please try again in a few minutes."]. Reproduced 8 times across two projects (471abde1-2a61-4083-9c40-01863bb5639e service mcr-worker, and a freshly created project aa9f253f-e122-4ba7-8591-1a79c3325f8c), via CLI railway up on versions 5.30.4 and latest. Last successful deploy was 13:34 UTC today with the identical command. I upgraded my plan yesterday — possibly related. Deployment IDs: e0aaa7be, b96c4a13, 5b14b4f1, 976e76ba, 5c088cfb, be3235a5, 8a7697dc, 4827cd2c.
3 Replies
Status changed to Awaiting Railway Response Railway • 17 days ago
17 days ago
I think I found it. The archive your CLI uploads is malformed. Two files, fonts/opensans600.ttf and fonts/atkinson400.ttf, are listed at their full size (122KB and 52KB) but contain no data at all. This causes our builder to fail when unpacking your files.
Most likely they're symlinks, or placeholder files that never fully downloaded locally. Can you check them directly, with wc -c on both files and ls -lL on the fonts folder, and confirm both are real files with actual bytes in them? Once they are, run railway up again and it should go straight through.
Status changed to Awaiting User Response Railway • 17 days ago
16 days ago
Checked both, and they're real files with real bytes. wc -c returns 122,060 for opensans600.ttf and 52,476 for atkinson400.ttf, matching the sizes you saw. ls -lL and ls -l are identical with mode -rw-r--r--, so no symlinks. file reads both as valid TrueType (18 and 17 tables), 104,628 and 42,815 of those bytes are non-zero, both have real data in their final 16 bytes, and neither is sparse (240 and 104 allocated blocks). PIL loads both and renders glyphs — "Open Sans SemiBold" and "Atkinson Hyperlegible Regular". They're byte-identical to their git blobs, and we don't use LFS. I also tarred, gzip-tested, and re-extracted the folder: both come back md5-identical.
The part that I think rules them out entirely: those two files last changed on Jul 31, and my last successful deploy was Aug 4 at 13:34Z. The same bytes unpacked fine on your builder four days after they reached their current state, and no font changed between that success and the nine failures that followed.
Given that, could you check whether the stored archive's byte length matches what the CLI reported uploading? Two arbitrary entries carrying a header size with no payload reads to me like the snapshot step truncating the stream rather than bad input. Happy to run anything else that would help.
Status changed to Awaiting Railway Response Railway • 16 days ago
Status changed to Awaiting Conductor Response brody • 16 days ago
16 days ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 16 days ago
mykal
I think I found it. The archive your CLI uploads is malformed. Two files, `fonts/opensans600.ttf` and `fonts/atkinson400.ttf`, are listed at their full size (122KB and 52KB) but contain no data at all. This causes our builder to fail when unpacking your files. Most likely they're symlinks, or placeholder files that never fully downloaded locally. Can you check them directly, with `wc -c` on both files and `ls -lL` on the fonts folder, and confirm both are real files with actual bytes in them? Once they are, run railway up again and it should go straight through.
15 days ago
I checked, doesnt apply to my enviroment. deployment was normal as well