19 hours ago
Deployed container doesn't match the image that was actually built (Dockerfile builder)
I'm hitting something that looks like a platform-side bug: my service crashes at runtime with an error that should be structurally impossible given what the build logs show was actually installed.
The crash
File ".../charset_normalizer/api.py", line 5, in <module>
from .cd import (
File "src/charset_normalizer/cd.pyx", line 1, in init charset_normalizer.cd
AttributeError: module 'charset_normalizer.md' has no attribute 'CharInfo'This happens on import requests at app startup (a transitive dependency pulls in charset_normalizer).
What I tried, and what it proved
- Pinned various versions of the offending package (3.4.4, 3.4.3, 3.3.2). Same crash every time, regardless of version.
- Forced a from-source rebuild (
--no-binary,--force-reinstall) instead of trusting a prebuilt wheel — produces a verified pure-Python wheel with no compiled extension at all (confirmed vialsin the build log: only.pyfiles, no.so/.pyx). Same crash. - Wiped all
__pycache__/.pycfiles after install, in case of stale bytecode. Same crash. - Added a build-time diagnostic step that runs
python3 -c "import charset_normalizer"immediately after installing it, inside the Dockerfile itself — this printed a clean success message every time, and confirmed viafind /that only one copy of the package exists anywhere in the image, with no compiled artifacts. - Replicated the exact application import chain that crashes at runtime, as a build-time step — also succeeded cleanly at build time.
- Forced a guaranteed cache-bust (a unique, never-before-seen string inserted directly into the Dockerfile right before the install step) to rule out any layer/cache deduplication. Same crash.
- Confirmed the same crash from a real git-push-triggered build (not just CLI-triggered deploys), ruling out anything CLI-specific.
- Removed the package entirely (
pip uninstall), confirmed via the build log that it was successfully uninstalled and is not present in the image. The runtime traceback still shows this exact package being imported and crashing.
That last point is the one I can't explain: a package that doesn't exist in the built image is still being imported and crashing at runtime. Every single build was independently verified clean via in-build diagnostics, immediately before the resulting image reproduced the identical crash at runtime.
Also ruled out
- A custom Start Command override at the service level (confirmed it's not related to the crash itself, just something that initially made Dockerfile
CMDchanges look like they weren't taking effect). - Wrong deploy branch — confirmed the correct branch is what's configured and being built.
- A second/shadow Python interpreter in the image (only one
python3, onerequests/charset_normalizerlocation, confirmed viadpkg,which -a, andfind /).
Has anyone else seen a case where the container actually serving traffic doesn't match what the build logs say was produced? Happy to share the Dockerfile / more detail if useful — just stripped identifying project details for this post.
2 Replies
18 hours ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 19 hours ago
2 hours ago
This is one of the more thoroughly-ruled-out reports I've seen on here — you've already eliminated stale bytecode, layer caching, CLI-vs-git-push deploy paths, and even verified via find / that only one clean copy exists in the built image. Given that, I don't think this is a requirements.txt/dependency issue at all anymore — worth pointing a couple of things at Railway directly rather than the app code:
- Is this related to the active "Deployments slow to start" incident? Given the timing and the fact that your symptom is literally "the container serving traffic doesn't match what was built," this smells like it could be a builder→registry→deploy propagation issue — i.e. the deploy step pulling a stale/cached image reference instead of the one that was just verified clean. Worth asking Railway explicitly whether these are connected.
- One thing not fully ruled out yet: if your Dockerfile uses a multi-stage build, it's worth double-checking that your build-time diagnostic (python3 -c "import charset_normalizer") runs in the exact same final stage that gets deployed — not an earlier builder stage whose output then gets COPY --from= into a final stage that could theoretically reuse a cached layer independently. Your cache-bust string proves the builder stage isn't cached, but if there's a separate final stage, that's a distinct cache domain.
- A more direct way to prove it either way: instead of a build-time check, add the diagnostic to your actual runtime start command (or the very first line your app runs) — e.g. print pip freeze or hash the charset_normalizer package directory at container startup, right before the crash happens. If that output doesn't match what your build log showed, that's airtight proof the running container's filesystem differs from what was built — which is a Railway platform bug, not anything on your end. If Railway support sees that, it should be a fast escalation. This really does need someone with access to the builder/registry/deploy pipeline internals to confirm — but the runtime-hash test above would give you (and them) undeniable evidence either way.
dolapobayo42-lgtm
This is one of the more thoroughly-ruled-out reports I've seen on here — you've already eliminated stale bytecode, layer caching, CLI-vs-git-push deploy paths, and even verified via find / that only one clean copy exists in the built image. Given that, I don't think this is a requirements.txt/dependency issue at all anymore — worth pointing a couple of things at Railway directly rather than the app code: 1. Is this related to the active "Deployments slow to start" incident? Given the timing and the fact that your symptom is literally "the container serving traffic doesn't match what was built," this smells like it could be a builder→registry→deploy propagation issue — i.e. the deploy step pulling a stale/cached image reference instead of the one that was just verified clean. Worth asking Railway explicitly whether these are connected. 2. One thing not fully ruled out yet: if your Dockerfile uses a multi-stage build, it's worth double-checking that your build-time diagnostic (python3 -c "import charset_normalizer") runs in the exact same final stage that gets deployed — not an earlier builder stage whose output then gets COPY --from= into a final stage that could theoretically reuse a cached layer independently. Your cache-bust string proves the builder stage isn't cached, but if there's a separate final stage, that's a distinct cache domain. 3. A more direct way to prove it either way: instead of a build-time check, add the diagnostic to your actual runtime start command (or the very first line your app runs) — e.g. print pip freeze or hash the charset_normalizer package directory at container startup, right before the crash happens. If that output doesn't match what your build log showed, that's airtight proof the running container's filesystem differs from what was built — which is a Railway platform bug, not anything on your end. If Railway support sees that, it should be a fast escalation. This really does need someone with access to the builder/registry/deploy pipeline internals to confirm — but the runtime-hash test above would give you (and them) undeniable evidence either way.
18 minutes ago
Update: found the fix, thanks to your suggestion above.
Cause, confirmed by hashing every file in the package's install directory both at build time and at actual container startup: the .py source files matched byte-for-byte between build and runtime, but the running container also had extra compiled files (cd.so, md.so, md__mypyc.so, plus pycache/*.pyc I'd explicitly deleted during the build) that simply didn't exist anywhere in the built image. Something in the platform is injecting those on top of a correctly-built container, and the injected .so doesn't match the .py it expects — hence the AttributeError. Not fixable from the build side, since removing the package from the build entirely didn't stop the broken files from reappearing at runtime.
Fix: instead of fighting what's on disk, block the import at the Python level — sys.modules.setdefault("charset_normalizer", None) early in app startup. Python's import system treats a None entry in sys.modules as "block this module," so import charset_normalizer raises a clean ImportError without ever touching the filesystem, regardless of what gets injected there. requests already catches that ImportError internally and falls back to skipping automatic charset detection (which we don't need — every response we parse already declares its own encoding). Verified this works even with the package freshly installed and physically present on disk. Deployed and confirmed stable, serving real traffic now.
Thanks again for pointing me at the runtime-hash approach instead of build-time — that was the piece that actually cracked it.