Builds running very slowly and typically crashing
vincentn-waivern
PROOP

16 days ago

Hi

My application stack is building extremely slowly (now 30 mins+) and has crashed several time. Here are log files from crashes. The stack is summarised below.

Stack:

  • Runtime: Python 3.12 (FastAPI + Uvicorn)

  • Browser automation: Playwright with headless Chromium for live website scanning

  • AI analysis: Claude API (Anthropic) for UX screenshot assessment, cookie classification, legal analysis, and policy link detection

  • Word report generation: Node.js + docx-js library (installed via apt in the container)

  • Container: Docker, based on mcr.microsoft.com/playwright/python:v1.49.1-noble (Ubuntu Noble)

  • No database: Runs and reports stored as JSON/HTML/DOCX files on a Railway volume mounted at /data

  • Two instances: EU (Amsterdam, consent-analyser-emea.waivern.com) and US (California, consent-analyzer-us.waivern.com)

Attachments

Solved

6 Replies

Status changed to Awaiting Railway Response Railway 16 days ago


vincentn-waivern
PROOP

16 days ago

The EMEA instance build which had hung for about 40 minutes suddenly completed! Did you fix something?


16 days ago

No changes were made on our end. Build times can vary due to resource availability on our build machines, and cache hits are not guaranteed since the build system scales dynamically. Your Playwright base image (mcr.microsoft.com/playwright/python:v1.49.1-noble) is very large, which is the main contributor to long build times. Reducing the container size, for example, using a multi-stage build or a smaller base image, would help. More on that at docs.railway.com/builds/build-configuration.

Regarding crashes, can you please share specific errors?


Status changed to Awaiting User Response Railway 16 days ago


vincentn-waivern
PROOP

16 days ago

These are all the error logs I received today for the deploys that failed. There is another deploy still stalled on the US server. Here are the current build logs from that, now at 45 mins and counting. Before this morning, builds were typically taking 2-3 minutes.

[Region: us-west1]

=========================

Using Detected Dockerfile

=========================

context: kmdv-bAfa

internal

load build definition from Dockerfile

0ms

internal

load metadata for mcr.microsoft.com/playwright/python:v1.49.1-noble

470ms

internal

load .dockerignore

0ms

1

FROM mcr.microsoft.com/playwright/python:v1.49.1-noble@sha256:42e251ec3238672c931be6c3565ff279690ae072c3e22058128d0412494f6866

8ms

internal

load build context

0ms

2

WORKDIR /app

96ms

3

COPY requirements.txt .

18ms

4

RUN pip install --no-cache-dir -r requirements.txt

5s

[notice] To update, run: python -m pip install --upgrade pip

5

RUN for i in 1 2 3; do rm -rf /var/lib/apt/lists/* && apt-get update && break || echo "apt-get update attempt $i failed, retrying..." && sleep 5; done && playwright install chromium --with-deps

7m 53s

Processing triggers for libc-bin (2.39-0ubuntu8.3) ...

6

RUN for i in 1 2 3; do rm -rf /var/lib/apt/lists/* && apt-get update && break || echo "apt-get update attempt $i failed, retrying..." && sleep 5; done && apt-get install -y --no-install-recommends nodejs npm && npm install -g docx && rm -rf /var/lib/apt/lists/*

36m 58s

Get:143 http://archive.ubuntu.com/ubuntu noble/universe amd64 node-indent-string all 4.0.0-2 [4122 B]


Status changed to Awaiting Railway Response Railway 16 days ago


16 days ago

Sorry, I am not seeing errors here. Without that, we unfortunately won't be able to assist you here.


Status changed to Awaiting User Response Railway 16 days ago


vincentn-waivern
PROOP

16 days ago

OK. Sudden transition to 20x longer deploy times is not a problem in your humble opinion, then. Hmmm..... I'll try to follow the playwright advice you gave above, and see if it makes any diff.


Status changed to Awaiting Railway Response Railway 16 days ago


16 days ago

To be clear, we're not saying this isn't a problem. A jump from 2-3 minutes to 40+ minutes is disruptive and we understand the frustration. Unfortunately it is a side effect of how our build infrastructure works at this time. Optimizing your Dockerfile to reduce the work done in uncached builds is the most reliable way to keep build times consistently low.


Status changed to Awaiting User Response Railway 16 days ago


Railway
BOT

8 days ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway 8 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...