update: Following the earlier ldd result (both libgobject-2.0.so.0 and libglib-2.0.so.0 resolve every dependency cleanly via ldd, while ctypes.CDLL() of the exact same absolute path fails from inside our Python process), we checked the process's security state directly from /proc/self/status:

uid/euid/suid: 0/0/- gid/egid: 0/0

CapEff: 00000000800405fb CapBnd: 00000000800405fb

NoNewPrivs: 0 Seccomp: 2 Seccomp_filters: 3

UID/GID mismatch (glibc "secure execution" mode) is ruled out: uid == euid == 0, gid == egid. No mismatch, so glibc wouldn't be dropping LD_LIBRARY_PATH for this reason.

Seccomp restriction is also ruled out as the differentiator: a filter is active (mode 2, 3 stacked filters), but the ldd subprocess that succeeded was spawned as a child of this exact process — seccomp filters are inherited and can only get stricter in children, never looser. So ldd ran under the identical (or stricter) filter set and still succeeded at the equivalent operation (loading and resolving the library). If a seccomp rule were blocking a syscall dlopen() needs, ldd would have hit it too.

With both of those cleared, the only remaining difference between "resolves fine via ldd" and "fails via ctypes.CDLL/weasyprint's cffi-based loader" is something specific to how cffi itself invokes the native dynamic loader from inside an already-running Python process — not the filesystem, not the library, not the search path, and not a sandbox/security restriction we can detect. This is likely at the level of cffi's internal loading mechanism (e.g. an isolated loader namespace) rather than anything in our own configuration.

Nothing really for us to do I think. Any pointers appreciated

ldd succeeded because it's a child process. That's the difference.

You set LD_LIBRARY_PATH from Python at import time. That does nothing to the running process — glibc reads it once at exec and caches the search list. But subprocess passes os.environ to children, so ldd started with your prepended path and found the library. Your Python process never had it.

That's why ldd resolves and CDLL on the same absolute path fails.

Check it from inside the failing request:

open('/proc/self/environ','rb').read().split(b'\0') # env at exec

os.environ['LD_LIBRARY_PATH'] # your edited copy

/proc/self/environ doesn't change when Python calls putenv. If those disagree, the loader used the first one.

And:

env -u LD_LIBRARY_PATH ldd /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0

Should now fail on libglib-2.0.so.0, same as your process does.

Two other things:

ctypes.CDLL and cffi both call plain dlopen(). Isolated namespaces need dlmopen(), which neither uses — so that's not it.

ldconfig -p was never relevant. Nixpacks gives you a Nix-built Python (your path has /nix/store/…gcc-13.3.0-lib/lib), and nixpkgs patches glibc to read a store-local ld.so.cache instead of /etc/ld.so.cache.

Fix: set LD_LIBRARY_PATH as a Railway service variable so it exists before the process starts. Or install glib/pango/cairo/gdk-pixbuf via nixPkgs instead of aptPkgs so they're RPATH-resolved. Or use a Dockerfile.

Thank you for the guidance. Problem resolved.

The library file existing on disk, being correctly cached by ldconfig, and being on LD_LIBRARY_PATH were all real, but none of them were the actual problem. The real cause: python311 was installed via Nixpacks' nixPkgs, so the venv's python binary is a Nix-built binary linked against Nix's own patched glibc via RUNPATH — not Ubuntu's system glibc, even though the apt-installed native libraries (pango/cairo/glib) live in the normal /usr/lib/x86_64-linux-gnu.

Putting that multiarch directory on LD_LIBRARY_PATH looked like the obvious fix, and it even correctly resolved weasyprint's own libraries — but LD_LIBRARY_PATH takes priority over RUNPATH for every dynamic load in the process, including the interpreter's own dependency on glibc itself. That broke the Nix-built python binary before it even started (error while loading shared libraries: __vdso_gettimeofday: invalid mode for dlopen()) — a worse failure than the original problem, since it took the whole process down instead of leaving a working fallback.

The fix that worked, with zero LD_LIBRARY_PATH risk: preload each native library by absolute path via ctypes.CDLL(path, mode=ctypes.RTLD_GLOBAL) before the library that needs them (weasyprint, via cffi) ever imports. glibc's dlopen() always checks whether a library with a matching soname is already resident in the process before doing any path search — so the later bare-name dlopen() calls find them already loaded and never search at all. This never touches the environment, so it can't affect the interpreter's own resolution.

One more thing worth flagging for anyone trying this: don't hand-list which libraries to preload. A hand-written list only ever surfaces one missing dependency per attempt (fixed libgobject → failure moved to libglib → fixed that → failure moved to libpango, which needed libfribidi/libthai that were never in the list). Running ldd against each top-level library and preloading the full union of what it resolves — since ldd correctly resolves the whole transitive closure in one pass in this environment — fixed it completely in one shot.

Best practice for Nixpacks + a language runtime from nixPkgs + native libs from aptPkgs: never add the apt library directory to LD_LIBRARY_PATH if your runtime itself came from nixPkgs. Treat that as a hard rule, not a first thing to try — it can silently break the interpreter for a completely unrelated reason. Preload via absolute path instead.

Thanks to @travismcelfresh who pinpointed the actual LD_LIBRARY_PATH-timing mechanism — that was the piece that made the rest of this tractable.

Nice work!