project: 45c1876a-a454-444e-9b70-e33bd5b762dd

https://pycairo.readthedocs.io/en/latest/getting_started.html
hmmm, it looks like pycairo needs libcairo2-dev pkg-config python3-dev

I dont know if nixpacks has this.
May I suggest using a custom Dockerfile?
You get much a higher level of control over anything when you use one. They might seem a bit odd at first but once you get the hang its wonderful.

we literally haven't changed anything today, and now we cannot deploy. It would take us quite a bit of time to build a custom Dockerfile now and this is literally blocking production deployments right now

Completely understand the frustration.
Let me do a bit of digging on this and get back to you

Can you share with me your current nixpacks.toml?

currently it's this as I tried to get it to find the package by forcing it but the error has been the same all along:

[phases.setup]
nixPkgs = ["sqlcipher", "coreutils", "openssh", "gnutar", "python3Packages.supervisor", "nginx", "gettext", "cairo", "pkg-config", "pango"]

[phases.install]
cmds = [
"python -m venv --copies /opt/venv && . /opt/venv/bin/activate && export PKGCONFIGPATH=\"$PKGCONFIGPATH:/root/.nix-profile/lib/pkgconfig\" && pip install -r requirements.txt"
]

(ie. we didn't use a custom phases.install earlier)

it looks like in setup its looking for pkgconfig not pkg-config

might be completely missing the point (not crazy familiar with nixpacks myself) but would swapping that or adding it to setup fix?

Let me know if it was that simple haha. The sheer number of times something this simple has happened is insane

no, the package is pkg-config.. it completely fails if I add pkgconfig

Sounds good. Wanted to rule that out haha.
Doing a bit more investigating

hmm.. I just noticed that all of our services builder has been changed to Railpack from Nixpacks

I remember seeing an email that it would be the default builder but we have not changed it on any of our projects

and it's not changing it back to nixpack no matter what I do

I change the builder, hit deploy and it changes it back to Railpack

Ok so here's what's happening:

1. The Settings UI always shows railpack as the builder but actually it's not

2. Nixpack builder was broken somehow today

3. I got the build to work with railpack

looks like someone's made an oopsie

did you guys just drop the ball on this one??

Hey there!
Just an FYI the community help is entirely volunteer driven.
I have been working an 18 hour day so far and havent had time to work through these 😁

Will try to get back when im fully able to think about it but somebody else might be in that time!

hey no worries man, I didn't know that. This is clearly a Railway error and should not be handled by a volunteer but Railway engineering.

Something has been broken in the builders.. the only fix was to switch to railpack as a builder. This makes me lose a bit of faith in Railway as a production platform.

@jmck0644 did you run into any trouble using railpack?

That's the new recommended builder (it's the default going forward).

no troubles.. it's just the fact that we have 50+ deployments with different versions etc so being forced to switch overnight is not a small feat

@jmck0644 I'm going to follow up here…

sh -c 'echo deb http://deb.debian.org/debian bookworm-backports main > /etc/apt/sources.list.d/backports.list && apt-get update && apt-get install -y -t bookworm-backports sqlcipher=4.6.1-2~bpo12+1 && node server.js'

Please note the node server.js at the end there- change that to your apps start command.

so there's no way to get it to actually install it in railpack.json? This seems a bit bonkers

but as a workaround I just took yout sh command and created a script out of that

At a glance at the docs, it looks like you could put that start command in railpack.json

"deploy": {
    "startCommand": "node dist/index.js"
  }

Can’t really find a way, Railway employee confirmed in the other thread as well

Ah then^

@Sam (Loudbook) Hey, I can help here: what are you running to validate that sqlcipher is running properly? i.e. what JS is running that actually uses this apt package?

What I'd like to do is setup a example project that replicates this problem and then determine the best flow to fix it. I just need some help understanding what the minimum repro of the issue looks like,

I believe the user was trying to use this package via SSH CLI? I’m not sure. All I know is they needed the package at runtime.

@jmck0644 if you have some more details here I can look into building in better support for this (via docs or changes to the core) in railpack.