Deployment crashed due to grpc import failure

jzy95310
FREE

2 months ago

Hi experts,

I'm trying to deploy my backend service on Railway with a customized build command. The build is successful, but the deployment crashed after that with the following deploy logs:

DEBUG: LD_LIBRARY_PATH setup

DEBUG: LD_LIBRARY_PATH=/nix/store/2d5spnl8j5r4n1s4bj1zmra7mwx0f1n8-xgcc-13.3.0-libgcc/lib:/nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib::/usr/lib

DEBUG: Testing grpc import...

Traceback (most recent call last):

  File "<string>", line 1, in <module>

  File "/nix/store/0flj08i381bfdfbrly8sk6vs36lfrhnb-python3-3.11.11/lib/python3.11/site-packages/grpc/__init__.py", line 22, in <module>

    from grpc import _compression

  File "/nix/store/0flj08i381bfdfbrly8sk6vs36lfrhnb-python3-3.11.11/lib/python3.11/site-packages/grpc/_compression.py", line 20, in <module>

    from grpc._cython import cygrpc

ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory

It seems that the C++ extension for grpcio is not corrected included. Below is my `nixpacks.toml` file:

[variables]
NODE_VERSION = "18"
PYTHON_VERSION = "3.11"
NIXPKGS_ALLOW_UNFREE = "1"

[phases.setup]
nixPkgs = [
    "nodejs_18", 
    "npm-9_x", 
    "python311",
    "python311Packages.pip",
    "python311Packages.aiohttp",
    "python311Packages.python-dotenv",
    "python311Packages.pydantic",
    "python311Packages.fastapi",
    "python311Packages.googlemaps",
    "python311Packages.httpx",
    "python311Packages.pyyaml",
    "python311Packages.tenacity",
    "python311Packages.uvicorn",
    "python311Packages.requests",
    "python311Packages.click",
    "python311Packages.starlette",
    "python311Packages.grpcio",
    "python311Packages.grpcio-status",
    "python311Packages.google-api-core",
    "python311Packages.google-auth",
    "python311Packages.google-generativeai",
    "stdenv.cc.cc.lib",
    "glibc",
    "zlib",
    "autoPatchelfHook"
]

[phases.install]
cmds = [
    "npm install",
    "cd web/frontend && npm install",
    "# TEMPORARY: MCP not available in Railway's nixpkgs version (ffeebf0acf3ae8b29f8c7049cd911b9636efd7e7)",
    "# Available in nixpkgs 25.05+ but Railway uses older version",
    "# TODO: Remove --break-system-packages when Railway updates nixpkgs",
    "pip3 install --break-system-packages mcp>=1.0.0",
    "# Install backend requirements",
    "cd web/backend && pip3 install --break-system-packages -r requirements.txt",
    "# Patch ELF binaries to fix library paths",
    "find /nix/store -name '*.so' -exec autoPatchelf {} \\; 2>/dev/null || true"
]

[phases.build]
cmds = [
    "cd web/frontend && NEXT_PUBLIC_SUPABASE_URL=${NEXT_PUBLIC_SUPABASE_URL:-https://xxx.supabase.co} npm run build"
]

[start]
cmd = "cd web/frontend && npm start"

And below is the file `railway-backend.toml` which contains the customized build commands:

[build]
builder = "NIXPACKS"
buildCommand = "cd web/backend && pip3 install --break-system-packages -r requirements.txt"

[deploy]
startCommand = "echo 'DEBUG: LD_LIBRARY_PATH setup' && export LD_LIBRARY_PATH=$(find /nix/store -name '*gcc*' -type d | head -1)/lib:$(find /nix/store -name '*glibc*' -type d | head -1)/lib:$LD_LIBRARY_PATH && echo \"DEBUG: LD_LIBRARY_PATH=$LD_LIBRARY_PATH\" && echo 'DEBUG: Testing grpc import...' && python3 -c 'import grpc; print(\"SUCCESS: grpc imported\")' && cd web/backend && python3 -m uvicorn app.main:app --host 0.0.0.0 --port $PORT"
restartPolicyType = "ON_FAILURE"
restartPolicyMaxRetries = 10

Really appreciate any help you can provide to resolve this error. Thanks a lot!

$10 Bounty

1 Replies

idiegea21
HOBBY

a month ago

Heyy
Use the updated startCommand below:

startCommand = """ export LIBSTDCPP_PATH=$(find /nix/store -name libstdc++.so.6 | head -n 1 | xargs dirname) && \ export GLIBC_PATH=$(find /nix/store -name 'libc.so.6' | head -n 1 | xargs dirname) && \ export LD_LIBRARY_PATH=$LIBSTDCPP_PATH:$GLIBC_PATH:$LD_LIBRARY_PATH && \ python3 -c 'import grpc; print("SUCCESS: grpc imported")' && \ cd web/backend && python3 -m uvicorn app.main:app --host 0.0.0.0 --port $PORT """

```

this will make sure that all dependacies especially libstdc++.so.6 (that caused the crash) is available at runtime

Deployment crashed due to grpc import failure - Railway Help Station