2 hours ago
I have several custom Docker containers that I build locally and then push to GHCR. Most of them work correctly on Railway, but I have noticed a strange issue with some images where my Dockerfile removes files from the base image.
The workflow I use is:
- Start with an existing Docker image as the base.
- Create a Dockerfile that modifies the image and copies my configuration files into it.
- Build the image locally.
- Test the resulting image locally.
- Push the image to GHCR.
- Configure a Railway service to use the GHCR image.
!!! I do not build the image on Railway. Railway only pulls the existing image from GHCR and creates a container from it. !!!
Now for a concrete example. I am trying to deploy an alloy container. I use the original alloy image as base and I create my image from it with a Dockerfile like this:
========================================================
FROM grafana/alloy:latest
RUN rm -f /etc/alloy/config.alloy
COPY *.alloy /etc/alloy/
EXPOSE 12345
EXPOSE 5424/udp
ENTRYPOINT ["alloy", "run"]
CMD [ "--server.http.listen-addr=0.0.0.0:12345", "--storage.path=/var/lib/alloy/data", "/etc/alloy/"]
========================================================
Now when I run this locally it works as expected and there is no config.alloy file as it was removed from the image. I do the steps I described earlier, but on railway the config.alloy exists. I confirmed this several times, because the deployment failed from a conflict between my config and the default config.alloy and then I checked it with just listing the files in /etc/alloy/ folder. There are no Railway volumes attached to the service and I have not configured a custom start command. The service is configured to use the GHCR image directly.
My question is what could cause this issue, because it seems it happens only on railway.
(Important note I do not build the image on railway, I just pull the existing image and create a container for it on the platform.)
5 Replies
2 hours ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 2 hours ago
2 hours ago
if you're pushing to a floating tag like :latest, that's likely it. railway caches the resolved digest behind a tag for hours rather than repulling every deploy, so it can still be running an older build of your image, one from before you added the rm line, while your local machine has the current one. same tag, different actual image, which fits exactly what you're seeing.
quick way to confirm: pin the service to the digest instead of the tag (ghcr.io/you/image@sha256:...) or push with a unique tag each time (:v1, :v2, or the git sha) and point the service at that. if the file disappears once you do, that was it.
docs on the caching behaviour: https://docs.railway.com/deployments/image-auto-updates
manuproject
if you're pushing to a floating tag like :latest, that's likely it. railway caches the resolved digest behind a tag for hours rather than repulling every deploy, so it can still be running an older build of your image, one from before you added the rm line, while your local machine has the current one. same tag, different actual image, which fits exactly what you're seeing. quick way to confirm: pin the service to the digest instead of the tag (ghcr.io/you/image@sha256:...) or push with a unique tag each time (:v1, :v2, or the git sha) and point the service at that. if the file disappears once you do, that was it. docs on the caching behaviour: https://docs.railway.com/deployments/image-auto-updates
2 hours ago
I have unique tags for each image push like :1.0.0, :1.0.1 for exactly this reason, I have tried to use the digest too hoping that it will solve my problem, but I got the same results, either way I will try it again.
I have unique tags for each image push like :1.0.0, :1.0.1 for exactly this reason, I have tried to use the digest too hoping that it will solve my problem, but I got the same results, either way I will try it again.
2 hours ago
if you're on an apple silicon mac, worth checking the architecture angle. you build and test locally on arm64, but railway runs amd64. if you're pushing multi-arch, those are two separately built variants under the same tag, and buildx caches layers per platform, so the amd64 one can be stale from before you added the rm line while the arm64 one you're testing is current.
test it directly:
docker run --rm --platform linux/amd64 ghcr.io/you/image:1.0.1 ls -la /etc/alloy
if config.alloy shows up there but not without the --platform flag, that's your answer, and rebuilding with --no-cache for amd64 specifically should clear it.
manuproject
if you're on an apple silicon mac, worth checking the architecture angle. you build and test locally on arm64, but railway runs amd64. if you're pushing multi-arch, those are two separately built variants under the same tag, and buildx caches layers per platform, so the amd64 one can be stale from before you added the rm line while the arm64 one you're testing is current. test it directly: docker run --rm --platform linux/amd64 ghcr.io/you/image:1.0.1 ls -la /etc/alloy if config.alloy shows up there but not without the --platform flag, that's your answer, and rebuilding with --no-cache for amd64 specifically should clear it.
an hour ago
I am building on a windows machine and the platform is linux/amd64, but I tried your suggestion anyway and there is no config.alloy present.
I am building on a windows machine and the platform is linux/amd64, but I tried your suggestion anyway and there is no config.alloy present.
an hour ago
the fact that it only affects images where you delete files from the base is probably the answer.
RUN rm doesn't actually remove the file, it stays in the base layer and yours just records a marker telling the runtime to hide it. docker applies that correctly locally, so it looks gone. if railway's image unpacking doesn't apply those markers, the file comes back, which would only ever affect images that delete something.
workaround: drop the rm line and just COPY your own file over /etc/alloy/config.alloy so it replaces the default instead of deleting it. if that works and the rm doesn't, that confirms it.