Files that deleted on Dockerfile still present on the container filesystem when deployed to Railways

vedmaka

HOBBYOP

2 months ago

There is an image that contains Apache. The image Dockerfile contains instructions that delete the following file during build:

/etc/apache2/sites-enabled/000-default.conf

When the image is deployed via Docker, the file is not present on the filesystem, as expected:

❯ docker run --rm -it ghcr.io/wikiteq/taqasta:1.43.5-20260122-dac4861 bash
root@a4c7b13609a8:/var/www/mediawiki/w# ls -la /etc/apache2/sites-enabled/
total 12
drwxr-xr-x 1 root root 4096 Jan 22 22:19 .
drwxr-xr-x 1 root root 4096 Jan 22 22:19 ..
-rw-r--r-- 1 root root 1291 Jan 22 22:16 mediawiki.conf

When the exactly same image is deployed to Railways the file IS present on the container filesystem:

Create new Docker image service
Use ghcr.io/wikiteq/taqasta:1.43.5-20260122-dac4861 as the image name
Use sleep infinity as the Startup command
SSH into the service using the railway CLI

railway ssh --project=xxx --environment=xxx --service=xxx

root@c122b5f5cbb8:/var/www/mediawiki/w# ls -la /etc/apache2/sites-enabled/
total 12
drwxr-xr-x 1 root root 4096 Jan 22 23:44 .
drwxr-xr-x 1 root root 4096 Jan 22 23:44 ..
lrwxrwxrwx 1 root root   35 Jan 14 14:00 000-default.conf -> ../sites-available/000-default.conf
-rw-r--r-- 1 root root 1291 Jan 22 23:42 mediawiki.conf

And is owned (has been created?) by www-data (33) user

The image tags in both cases are exactly the same ghcr.io/wikiteq/taqasta:1.43.5-20260122-dac4861
In both cases the container startup scripts are effectively disabled by overwriting the command to bash for Docker run and to sleep infinity for Railways
The file is not present in the image filesystem and is explicitly deleted by the Dockerfile as can be seen here https://github.com/WikiTeq/Taqasta/blob/ce8d2cb14ee94c32af9272dc11568b4ecda5999c/templates/base.Dockerfile#L102
In both examples NO volumes attached to the containers and no extra args or envs are supplied

I am scratching my head.. is there something beyond downloading the image from the registry going on under the hood of the Railway?

Any ideas are welcomed!

Please be free to use the steps mentioned above to reproduce this mystery yourself

---

UPD: seems related:

https://station.railway.com/questions/image-layers-with-deleted-files-are-not-d7d808ee

Awaiting User Response

10 Replies

angelo-railway

EMPLOYEE

2 months ago

I have a meta question, I am wondering why you are using the container fs vs. using a mounted filesystem or an object store?

Status changed to Awaiting User Response Railway • about 2 months ago

angelo-railway

I have a meta question, I am wondering why you are using the container fs vs. using a mounted filesystem or an object store?

vedmaka

HOBBYOP

2 months ago

This is not about storing files at runtime, this is about container filesystem extracted from image layers. Here is a minimal snippet to make it easier to understand:

FROM debian:12.5
RUN apt-get -y install apache2
RUN rm /etc/apache2/sites-enabled/000-default.conf # <--- the file is present on Railway

Status changed to Awaiting Railway Response Railway • about 2 months ago

vedmaka

HOBBYOP

2 months ago

Also please see an example posted here https://station.railway.com/questions/image-layers-with-deleted-files-are-not-d7d808ee#7kev as this seem to be a broader issue

vedmaka

HOBBYOP

2 months ago

Also linking my comment from here https://station.railway.com/questions/image-layers-with-deleted-files-are-not-d7d808ee#v6qw

angelo-railway

EMPLOYEE

2 months ago

All good and noted- I have tagged the ticket on that thread for investigation. I apologize for not taking the matter seriously at the start, but now since we saw the other examples we have enough to go by. It's escalated.