10 months ago
I have a small project, using so far:
Two GitHub deploys
One Redis instance
One MongoDB instance
I made some code changes to a FastAPI/Python deployment, that uses Docker and Dockerfiles to build. Up until today I had absolutely no issue deploying changes from GitHub; however, in the past 4 hours, my deployments have failed at least 10 times. Every time with a different error: last redeploy, I got an OCI error.
Needless to say, I can compile and run the Docker image & container on my local environment without any hassle.
I looked around and found no reason why this should fail so spectacularly. Please help!
22 Replies
10 months ago
Please provide build logs.
10 months ago
This is my latest build:
Using dockerignore file: /kaniko/snapshots/a03569d3-4b96-4344-9352-0935104767e8/.dockerignore
Retrieving image manifest python:3.12
Retrieving image python:3.12 from registry index.docker.io
Returning cached image manifest
Built cross stage deps: map[]
Returning cached image manifest
Returning cached image manifest
Executing 0 build triggers
Building stage 'python:3.12' [idx: '0', base-idx: '-1']
Checking for cached layer us-west1.registry.rlwy.net/b0bfc537-f787-4dae-8b51-3318878e297c/cache:b8f02b381bdddf091b663d8725a4ca61cd3d8fda568615312f9a8598e87f7fb5...
Using caching version of cmd: RUN pip install --no-cache-dir --upgrade -r /app/requirements.txt
Cmd: EXPOSE
Adding exposed port: 8080/tcp
Unpacking rootfs as cmd COPY ./requirements.txt /app/requirements.txt requires it.
WORKDIR /app
Creating directory /app with uid -1 and gid -1
Taking snapshot of files...
COPY ./requirements.txt /app/requirements.txt
Taking snapshot of files...
Found cached layer, extracting to filesystem
COPY . /app/
Taking snapshot of files...
Cmd: EXPOSE
No files changed in this command, skipping snapshotting.
CMD ["python", "main.py"]
No files changed in this command, skipping snapshotting.
Pushing image to us-west1.registry.rlwy.net/b0bfc537-f787-4dae-8b51-3318878e297c:4c5eb60a-a53e-47af-bb34-fc22817b8681
Pushed us-west1.registry.rlwy.net/b0bfc537-f787-4dae-8b51-3318878e297c@sha256:80775415bca9a702cd4561c5cbb9f8fdc3a14ee65abc93d85399e6ee8045351b
=========================
Container failed to start
=========================
crun: executable file `python main.py` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found
10 months ago
# Dockerfile
FROM python:3.12
WORKDIR /app
COPY ./requirements.txt /app/requirements.txt
RUN pip install --no-cache-dir --upgrade -r /app/requirements.txt
COPY . /app/
EXPOSE 8080
CMD ["python", "main.py"]
And
# .dockerignore
# folders
__pycache__/
venv/
tests/
# files
.tool-versions
Makefile
Procfile
*.md
10 months ago
Looks simply enough, nothing looks wrong.
Long shot, would you happen to have a PATH
service variable set?
10 months ago
Previous crash was even cooler and weirder, I've never seen an OCI error:
[Region: us-west1]
=========================
Using Detected Dockerfile
=========================
context: 095f1fe4ff7671be65363f3221d87e2a
#0 building with "default" instance using docker driver
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 232B done
#1 DONE 0.0s
#2 [internal] load metadata for docker.io/library/python:3.12
#2 DONE 0.5s
#3 [internal] load .dockerignore
#3 transferring context: 122B done
#3 DONE 0.0s
#4 [1/5] FROM docker.io/library/python:3.12@sha256:4584ea46d313a10e849eb7c5ef36be14773418233516ceaa9e52a8ff7d5e35a5
#4 DONE 0.0s
#5 [2/5] WORKDIR /app
#5 CACHED
#6 [internal] load build context
#6 transferring context: 44.21kB done
#6 DONE 0.0s
#7 [3/5] COPY ./requirements.txt /app/requirements.txt
#7 DONE 0.0s
#8 [4/5] RUN pip install --no-cache-dir --upgrade -r /app/requirements.txt
#8 16.62 Successfully installed ...
#8 16.62 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#8 16.84
#8 16.84 [notice] A new release of pip is available: 24.0 -> 24.1
#8 16.84 [notice] To update, run: pip install --upgrade pip
#8 DONE 17.7s
#9 [5/5] COPY . /app/
#9 DONE 0.0s
#10 exporting to image
#10 exporting layers
#10 exporting layers 1.0s done
#10 writing image sha256:acb315253f53f1016a4cfa811d710f59eb5717ddef85a585f78ccdb736632ca3 done
#10 naming to us-west1.registry.rlwy.net/b0bfc537-f787-4dae-8b51-3318878e297c:fec12eec-1ed3-4c87-8a04-bd6505fa0de1 done
#10 DONE 1.0s
Build time: 19.55 seconds
================
Publishing Image
================
The push refers to repository [us-west1.registry.rlwy.net/b0bfc537-f787-4dae-8b51-3318878e297c]
Preparing e548c9cb2c31
Pushed 1be284a6fc27
fec12eec-1ed3-4c87-8a04-bd6505fa0de1: digest: sha256:1b0ff133827437d018a01a31155def90310747155b8eeb2b65e8a6a29f4fdf66 size: 2841
Publish time: 8.36 seconds
=========================
Container failed to start
=========================
crun: unknown version specified: OCI runtime error
10 months ago
Looks simply enough, nothing looks wrong.
Long shot, would you happen to have a
PATH
service variable set?
I do not. Not on this deployment, anyway. I do have the recommender PORT
variable, and nothing resembling "PATH".
10 months ago
This error log is from the legacy builder that uses buildx, the first build logs you sent are from the new builder that uses kaniko.
What do you build locally with?
10 months ago
I want to say `docker build`?
10 months ago
Unfortunately, no. It's on a private org on GitHub. Is there anything you'd like to see I can share over here?
10 months ago
I'm looking for a way to reproduce this issue, could you create a minimal reproducible repo without anything sensitive in it that still shows this error?
10 months ago
Will do. Thanks for your assistance! I'll reply back with the public repo in a few
10 months ago
This is a bare-bones GitHub repo with the FastAPI app: https://github.com/DarkestAbed/visadoTest. I should mention I changed the Python img to alpine
, to see if it made any difference. It did not.
I tested the deployment locally, using docker build -t <tag> <imgName>
:
docker build -t visado-back .
[+] Building 0.8s (10/10) FINISHED docker:rootless
=> [internal] load build definition from Dockerfile
=> => transferring dockerfile: 239B
=> [internal] load metadata for docker.io/library/python:3.12-alpine
=> [internal] load .dockerignore
=> => transferring context: 122B
=> [1/5] FROM docker.io/library/python:3.12-alpine@sha256:44a5b997abb8de1ea47a4a91a81431f8dc8f789e2bf45530aa0db85344d70bad
=> [internal] load build context
=> => transferring context: 4.34kB
=> CACHED [2/5] WORKDIR /app
=> CACHED [3/5] COPY ./requirements.txt /app/requirements.txt
=> CACHED [4/5] RUN pip install --no-cache-dir --upgrade -r /app/requirements.txt
=> CACHED [5/5] COPY . /app/
=> exporting to image
=> => exporting layers
=> => writing image sha256:77047f10c1f953f8646bc1218ce3815561dd51c3bf6ebcd6550866337ee04de7
=> => naming to docker.io/library/visado-back
docker run --rm --name visado-back --publish 8080:8080 visado-back
INFO: Started server process [1]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
^CINFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [1]
I have not deployed it on Railway as of now.
10 months ago
Would you be so kind as to share your deploy settings?
10 months ago
Straight deploy from GitHub, worked the first time, then I enabled the new builder and it still worked, no service variables or anything else was changed or added.
10 months ago
I'll recreate my deployment to see if it works that way. Thanks for your assistance!
10 months ago
I was able to recreate the service and spin it up without any issue. For future references, that's what I'll do when I encounter a similar error. Still, I think it's a pretty weird behavior.
Thanks for everything!
Status changed to Solved railway[bot] • 10 months ago
10 months ago
There must have been something missconfigured on the service itself, either way, glad that you found a solution!