container crashed
drawingcontext
HOBBYOP

2 years ago

container worked and everything was working well but container crashed...not sure why..Here's the projectid - 1b883543-ff8a-4389-9f72-1fe4b7b73668

Closed

161 Replies

2 years ago

Is there anything in the logs that might show why it crashed?


drawingcontext
HOBBYOP

2 years ago

/bin/bash: line 1: web:: command not found

container event container restart

/bin/bash: line 1: web:: command not found

container event container died


drawingcontext
HOBBYOP

2 years ago

Also when the app was up, it timedout on few api calls and I tried web: gunicorn app:app --timeout 120


drawingcontext
HOBBYOP

2 years ago

but I am back to web: gunicorn app:app in procfile


2 years ago

web: is a Procfile thing, you should be able to set your start command to just gunicorn app:web --timeout 120

Railway hasn't supported procfiles in a long time


drawingcontext
HOBBYOP

2 years ago

File "/opt/venv/lib/python3.11/site-packages/gunicorn/util.py", line 402, in import_app

app = getattr(mod, name)

      ^^^^^^^^^^^^^^^^^^

AttributeError: module 'app' has no attribute 'web'

Failed to find attribute 'web' in 'app'.

[2024-09-13 14:09:51 +0000] [4] [INFO] Worker exiting (pid: 4)

[2024-09-13 14:09:51 +0000] [1] [INFO] Shutting down: Master

[2024-09-13 14:09:51 +0000] [1] [INFO] Reason: App failed to load.

container event container restart

[2024-09-13 14:09:52 +0000] [1] [INFO] Starting gunicorn 20.1.0

[2024-09-13 14:09:52 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)

[2024-09-13 14:09:52 +0000] [1] [INFO] Using worker: sync

[2024-09-13 14:09:52 +0000] [4] [INFO] Booting worker with pid: 4

Traceback (most recent call last):

File "/opt/venv/lib/python3.11/site-packages/gunicorn/util.py", line 402, in import_app

app = getattr(mod, name)

      ^^^^^^^^^^^^^^^^^^

AttributeError: module 'app' has no attribute 'web'

Failed to find attribute 'web' in 'app'.

[2024-09-13 14:09:52 +0000] [4] [INFO] Worker exiting (pid: 4)

[2024-09-13 14:09:52 +0000] [1] [INFO] Shutting down: Master

[2024-09-13 14:09:52 +0000] [1] [INFO] Reason: App failed to load.

container event container died

[2024-09-13 14:09:53 +0000] [1] [INFO] Starting gunicorn 20.1.0

[2024-09-13 14:09:53 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)

[2024-09-13 14:09:53 +0000] [1] [INFO] Using worker: sync

[2024-09-13 14:09:53 +0000] [4] [INFO] Booting worker with pid: 4

Traceback (most recent call last):

File "/opt/venv/lib/python3.11/site-packages/gunicorn/util.py", line 402, in import_app

app = getattr(mod, name)

      ^^^^^^^^^^^^^^^^^^

AttributeError: module 'app' has no attribute 'web'

Failed to find attribute 'web' in 'app'.

[2024-09-13 14:09:53 +0000] [4] [INFO] Worker exiting (pid: 4)

container event container restart

[2024-09-13 14:09:54 +0000] [1] [INFO] Shutting down: Master

[2024-09-13 14:09:54 +0000] [1] [INFO] Reason: App failed to load.

[2024-09-13 14:09:55 +0000] [1] [INFO] Starting gunicorn 20.1.0

[2024-09-13 14:09:55 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)

[2024-09-13 14:09:55 +0000] [1] [INFO] Using worker: sync

[2024-09-13 14:09:55 +0000] [4] [INFO] Booting worker with pid: 4

Traceback (most recent call last):

File "/opt/venv/lib/python3.11/site-packages/gunicorn/util.py", line 402, in import_app

app = getattr(mod, name)

      ^^^^^^^^^^^^^^^^^^

AttributeError: module 'app' has no attribute 'web'

Failed to find attribute 'web' in 'app'.

[2024-09-13 14:09:55 +0000] [4] [INFO] Worker exiting (pid: 4)

[2024-09-13 14:09:55 +0000] [1] [INFO] Shutting down: Master

[2024-09-13 14:09:55 +0000] [1] [INFO] Reason: App failed to load.

container event container died

[2024-09-13 14:09:56 +0000] [1] [INFO] Starting gunicorn 20.1.0

[2024-09-13 14:09:56 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)

[2024-09-13 14:09:56 +0000] [1] [INFO] Using worker: sync

[2024-09-13 14:09:56 +0000] [4] [INFO] Booting worker with pid: 4

Traceback (most recent call last):

File "/opt/venv/lib/python3.11/site-packages/gunicorn/util.py", line 402, in import_app

app = getattr(mod, name)

      ^^^^^^^^^^^^^^^^^^

AttributeError: module 'app' has no attribute 'web'

Failed to find attribute 'web' in 'app'.

[2024-09-13 14:09:56 +0000] [4] [INFO] Worker exiting (pid: 4)

[2024-09-13 14:09:56 +0000] [1] [INFO] Shutting down: Master

[2024-09-13 14:09:56 +0000] [1] [INFO] Reason: App failed to load.

container event container died

[2024-09-13 14:09:57 +0000] [1] [INFO] Using worker: sync


2 years ago

Can you share your repo? Your application might be called app and not web, in which case, you would use app:app


2 years ago

yes we do


drawingcontext
HOBBYOP

2 years ago

That is true.


drawingcontext
HOBBYOP

2 years ago

I am restarting with app:app


drawingcontext
HOBBYOP

2 years ago

Now the app is up. testing the api calls to see if it times out


drawingcontext
HOBBYOP

2 years ago

Same issue


drawingcontext
HOBBYOP

2 years ago

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): feed-api.cielo.finance:443

[2024-09-13 14<:16:1199656381486219374>51 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:4)

[2024-09-13 14<:16:1199656381486219374>51 +0000] [4] [INFO] Worker exiting (pid: 4)

[2024-09-13 14<:16:1199656381486219374>51 +0000] [5] [INFO] Booting worker with pid: 5


drawingcontext
HOBBYOP

2 years ago

It is a worker timeout


drawingcontext
HOBBYOP

2 years ago

Check the update


2 years ago

how long do you expect a single request to take


drawingcontext
HOBBYOP

2 years ago

it depends on the response but should not take more than 5-7 seconds tbh


2 years ago

when testing, how many RPS are you doing?


drawingcontext
HOBBYOP

2 years ago

what is RPS?


drawingcontext
HOBBYOP

2 years ago

locally it works


2 years ago

requests per second


drawingcontext
HOBBYOP

2 years ago

It is adata analysis application hitting many different apis


drawingcontext
HOBBYOP

2 years ago

so def. 100 req/sec


2 years ago

then you are likely running into issues with having run gunicorn with sync workers


2 years ago

start it with the uvicorn event worker


drawingcontext
HOBBYOP

2 years ago

how do I do it?


2 years ago

what's your current start command


drawingcontext
HOBBYOP

2 years ago

gunicorn app:app --timeout 120


drawingcontext
HOBBYOP

2 years ago

gunicorn app:app --timeout 120


2 years ago

no need to ping reply


2 years ago

what kind of app? fastapi, flask?


drawingcontext
HOBBYOP

2 years ago

flask


2 years ago

your start command would become gunicorn app:app --timeout 120 --worker-class eventlet

you will also need to make sure eventlet is in your requirements.txt file


drawingcontext
HOBBYOP

2 years ago

Here's my current req


drawingcontext
HOBBYOP

2 years ago

Flask==2.0.3

Werkzeug==2.0.3

Jinja2==3.0.3

itsdangerous==2.0.1

click==8.0.4

gunicorn==20.1.0

requests==2.26.0

prettytable==2.5.0

Flask-Login==0.5.0


drawingcontext
HOBBYOP

2 years ago

what version should I use for eventlet


2 years ago

the latest is probably fine


drawingcontext
HOBBYOP

2 years ago

ok, made changes and redeploying


drawingcontext
HOBBYOP

2 years ago

Error: class uri 'eventlet' invalid or not found:

[Traceback (most recent call last):

File "/opt/venv/lib/python3.11/site-packages/gunicorn/util.py", line 99, in load_class

mod = importlib.import_module('.'.join(components))

      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/root/.nix-profile/lib/python3.11/importlib/init.py", line 126, in import_module

return _bootstrap._gcd_import(name[level:], package, level)

       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "", line 1204, in _gcd_import

File "", line 1176, in _find_and_load

File "", line 1147, in _find_and_load_unlocked

File "", line 690, in _load_unlocked

File "", line 940, in exec_module

File "", line 241, in _call_with_frames_removed

File "/opt/venv/lib/python3.11/site-packages/gunicorn/workers/geventlet.py", line 20, in

from eventlet.wsgi import ALREADY_HANDLED as EVENTLET_ALREADY_HANDLED

ImportError: cannot import name 'ALREADY_HANDLED' from 'eventlet.wsgi' (/opt/venv/lib/python3.11/site-packages/eventlet/wsgi.py)

]

container event container died


2 years ago

do you have it in your requirements.txt?


drawingcontext
HOBBYOP

2 years ago

Flask==2.0.3

Werkzeug==2.0.3

Jinja2==3.0.3

itsdangerous==2.0.1

click==8.0.4

gunicorn==20.1.0

requests==2.26.0

prettytable==2.5.0

Flask-Login==0.5.0

eventlet==0.33.3


2 years ago

whats your start command


drawingcontext
HOBBYOP

2 years ago

shoujd I use gevent?


2 years ago

^


drawingcontext
HOBBYOP

2 years ago

gunicorn app:app --timeout 120 --worker-class eventlet


drawingcontext
HOBBYOP

2 years ago

I have a Procfile


drawingcontext
HOBBYOP

2 years ago

does it matter for Railway


2 years ago

no, we support procfiles


2 years ago

okay try gevent


drawingcontext
HOBBYOP

2 years ago

trying now


drawingcontext
HOBBYOP

2 years ago

invalid type: string "gunicorn --timeout 120 --worker-class gevent app:app", expected a ma


drawingcontext
HOBBYOP

2 years ago

did I get format right?


2 years ago

try moving app:app to after gunicorn


drawingcontext
HOBBYOP

2 years ago

context: 5f53c0eeaf434d6d998e9ed8ac0a2c0f

Nixpacks build failed

Error: Reading Procfile

Caused by:

invalid type: string "gunicorn app:app --timeout 120 --worker-class gevent", expected a map


drawingcontext
HOBBYOP

2 years ago

just way beyond my understanding...


drawingcontext
HOBBYOP

2 years ago

gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -I/nix/store/wwg6376ymkhpjxy6c3w9gkbhv6l2gmqy-libxcrypt-4.4.36/include -fPIC -DLIBEV_EMBED=1 -DEV_COMMON= -DEV_CLEANUP_ENABLE=0 -DEV_EMBED_ENABLE=0 -DEV_PERIODIC_ENABLE=0 -DEV_USE_REALTIME=1 -DEV_USE_MONOTONIC=1 -DEV_USE_FLOOR=1 -Isrc/gevent/libev -I/nix/store/k3701zl6gmx3la7y4dnflcvf3xfy88kh-python3-3.11.9/include/python3.11 -I/root/.nix-profile/include/python3.11 -I/tmp/pip-install-mht4g7p5/gevent_7b95d12f6294466681fc5ce8f6126ea3/deps -I/tmp/pip-install-mht4g7p5/gevent_7b95d12f6294466681fc5ce8f6126ea3/src/gevent/libev -I/tmp/pip-install-mht4g7p5/gevent_7b95d12f6294466681fc5ce8f6126ea3/deps/libev -Isrc/gevent -Isrc/gevent/libev -Isrc/gevent/resolver -I. -I/opt/venv/include -I/nix/store/k3701zl6gmx3la7y4dnflcvf3xfy88kh-python3-3.11.9/include/python3.11 -c src/gevent/libev/callbacks.c -o build/temp.linux-x86_64-cpython-311/src/gevent/libev/callbacks.o

#11 103.8 gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -I/nix/store/wwg6376ymkhpjxy6c3w9gkbhv6l2gmqy-libxcrypt-4.4.36/include -fPIC -DLIBEV_EMBED=1 -DEV_COMMON= -DEV_CLEANUP_ENABLE=0 -DEV_EMBED_ENABLE=0 -DEV_PERIODIC_ENABLE=0 -DEV_USE_REALTIME=1 -DEV_USE_MONOTONIC=1 -DEV_USE_FLOOR=1 -Isrc/gevent/libev -I/nix/store/k3701zl6gmx3la7y4dnflcvf3xfy88kh-python3-3.11.9/include/python3.11 -I/root/.nix-profile/include/python3.11 -I/tmp/pip-install-mht4g7p5/gevent_7b95d12f6294466681fc5ce8f6126ea3/deps -I/tmp/pip-install-mht4g7p5/gevent_7b95d12f6294466681fc5ce8f6126ea3/src/gevent/libev -I/tmp/pip-install-mht4g7p5/gevent_7b95d12f6294466681fc5ce8f6126ea3/deps/libev -Isrc/gevent -Isrc/gevent/libev -Isrc/gevent/resolver -I. -I/opt/venv/include -I/nix/store/k3701zl6gmx3la7y4dnflcvf3xfy88kh-python3-3.11.9/include/python3.11 -c src/gevent/libev/corecext.c -o build/temp.linux-x86_64-cpython-311/src/gevent/libev/corecext.o

#11 103.8 In file included from src/gevent/libev/libev.h:9,

#11 103.8 from src/gevent/libev/corecext.c:1319:

#11 103.8 /tmp/pip-install-mht4g7p5/gevent_7b95d12f6294466681fc5ce8f6126ea3/deps/libev/ev.c:580:48: warning: "/*" within comment [-Wcomment]

#11 103.8 580 | /#define MIN_INTERVAL 0.00000095367431640625 / 1/2**20, good till 2200 */

#11 103.8 |

#11 103.8 /tmp/pip-install-mht4g7p5/gevent_7b95d12f6294466681fc5ce8f6126ea3/deps/libev/ev.c: In function ‘ecb_binary32_to_binary16’:


drawingcontext
HOBBYOP

2 years ago

gone back to eventlet


drawingcontext
HOBBYOP

2 years ago

File "/opt/venv/lib/python3.11/site-packages/eventlet/greenio/base.py", line 32, in

socket_timeout = eventlet.timeout.wrap_is_timeout(socket.timeout)

                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/opt/venv/lib/python3.11/site-packages/eventlet/timeout.py", line 166, in wrap_is_timeout

base.is_timeout = property(lambda _: True)

^^^^^^^^^^^^^^^

TypeError: cannot set 'is_timeout' attribute of immutable type 'TimeoutError'

]

container event container died


drawingcontext
HOBBYOP

2 years ago

can I use python-3.9.7


2 years ago

yes, set it in a runtime.txt file


drawingcontext
HOBBYOP

2 years ago

I did that but no idea how to resolve the issue with timeouts


drawingcontext
HOBBYOP

2 years ago

neither gevent or eventlet is working on railway


2 years ago

is your app now using gevent


drawingcontext
HOBBYOP

2 years ago

gevent had build issues so I went back to eventlet


drawingcontext
HOBBYOP

2 years ago

but eventlet is crashing the docker


drawingcontext
HOBBYOP

2 years ago

Error: class uri 'eventlet' invalid or not found:


2 years ago

you woud need to get one of them working


drawingcontext
HOBBYOP

2 years ago

runtime.txt does not seem to be getting picked up


2 years ago

why do you say that


drawingcontext
HOBBYOP

2 years ago

File "/opt/venv/lib/python3.11


drawingcontext
HOBBYOP

2 years ago

using python3.11


2 years ago

i cant verify that, you are about to delete the service


drawingcontext
HOBBYOP

2 years ago

I deleted the service


drawingcontext
HOBBYOP

2 years ago

started a new one.


drawingcontext
HOBBYOP

2 years ago

and I got things back to a place where it is working


drawingcontext
HOBBYOP

2 years ago

not using eventlet or gevent


drawingcontext
HOBBYOP

2 years ago

just timeout


drawingcontext
HOBBYOP

2 years ago

I want to see if the error persists


drawingcontext
HOBBYOP

2 years ago

atleast now I don't have build issues or deployment issues


2 years ago

this wasnt an issue with railway so the timeout issue isnt going to change


drawingcontext
HOBBYOP

2 years ago

no timeouts locally


drawingcontext
HOBBYOP

2 years ago

local dev works fine


2 years ago

you are not running sync workers locally


2 years ago

are you sure this is flask?


drawingcontext
HOBBYOP

2 years ago

Yes


drawingcontext
HOBBYOP

2 years ago

app = Flask(name)


drawingcontext
HOBBYOP

2 years ago

same error


drawingcontext
HOBBYOP

2 years ago

[2024-09-13 16<:16:1199656381486219374>28 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:4)

[2024-09-13 16<:16:1199656381486219374>28 +0000] [4] [INFO] Worker exiting (pid: 4)

[2024-09-13 16<:16:1199656381486219374>28 +0000] [5] [INFO] Booting worker with pid: 5


2 years ago

you need to use an async worker


drawingcontext
HOBBYOP

2 years ago

that will require changes ot my code?


drawingcontext
HOBBYOP

2 years ago

I just want to replicate my local environment on railway


2 years ago

im not sure


2 years ago

in a production environment you must use gunicorn


drawingcontext
HOBBYOP

2 years ago

yes that works


2 years ago

and in your case you also need to use async workers


drawingcontext
HOBBYOP

2 years ago

but the moment I add eventlet, it is not even building


drawingcontext
HOBBYOP

2 years ago

the moment I add gevent, docker is crashing


2 years ago

try gthread then


drawingcontext
HOBBYOP

2 years ago

increasaed timeout sufficiently large so that works but towards the end, the final request the response got chopped off


drawingcontext
HOBBYOP

2 years ago

is there a limit on the size of response for railway?


2 years ago

no, only a 5 minute time limit


drawingcontext
HOBBYOP

2 years ago

this was the last step before rendering if you see logs


2 years ago

increasing the time-out here is just a Band-Aid solution


drawingcontext
HOBBYOP

2 years ago

still dont kknow why railway has issue.


drawingcontext
HOBBYOP

2 years ago

works fine on local


drawingcontext
HOBBYOP

2 years ago

works well on a vps server


drawingcontext
HOBBYOP

2 years ago

fails on railway


2 years ago

that not not indicate an issue with railway, that indicates an issue with your configurations on railway


2 years ago

you need to use an async worker so that your app can handle more then 3 requests concurrently


drawingcontext
HOBBYOP

2 years ago

yea, will need to implement celery and redis but was hoping that I can do that later.


drawingcontext
HOBBYOP

2 years ago

Maybe for now, I just use VPS and evaluate railway later on


2 years ago

you can do celery and redis later, right now you need to use an async worker


drawingcontext
HOBBYOP

2 years ago

didn't we try that?


drawingcontext
HOBBYOP

2 years ago

eventlet and gevent both crash the docker for whatever reason


2 years ago

you have something misconfigured and will need to look into why that's happening


drawingcontext
HOBBYOP

2 years ago

I'll modify the code to see if it works


drawingcontext
HOBBYOP

2 years ago

by importing asyncio & aiohttp and adding async with in front of all my request fetch functions


drawingcontext
HOBBYOP

2 years ago

Figured out a way.


drawingcontext
HOBBYOP

2 years ago

Is there a way to hard refresh a web page?


drawingcontext
HOBBYOP

2 years ago

feels like it is serving some cached version


2 years ago

railway doesnt do any asset caching, is it behind cloudflare?


drawingcontext
HOBBYOP

2 years ago

I migrated the Flask App to NodeJS app


drawingcontext
HOBBYOP

2 years ago

is the startup command - web: node app.js


drawingcontext
HOBBYOP

2 years ago

container is failing right now


drawingcontext
HOBBYOP

2 years ago

Build is successful


2 years ago

please provide the error


drawingcontext
HOBBYOP

2 years ago

/bin/bash: line 1: web:: command not found

container event container died


2 years ago

what is the contents of your procfile


drawingcontext
HOBBYOP

2 years ago

web: node app.js


2 years ago

I'll check back with this when I'm back at the computer so I can view your logs, I don't think you are sending me the correct error


drawingcontext
HOBBYOP

2 years ago

\

1285242638093778994


2 years ago

wait, aren't you deploying from a Dockerfile? why do you have a procfile


drawingcontext
HOBBYOP

2 years ago

I connected my github project from railway


drawingcontext
HOBBYOP

2 years ago

railway handles everything


drawingcontext
HOBBYOP

2 years ago

I can remove the procfile


drawingcontext
HOBBYOP

2 years ago

you want me to try that?


2 years ago

haha slow down there


drawingcontext
HOBBYOP

2 years ago

ok, let me know


drawingcontext
HOBBYOP

2 years ago

trying my luck ...


2 years ago

send me your Dockerfile please


drawingcontext
HOBBYOP

2 years ago

How do I do it?


drawingcontext
HOBBYOP

2 years ago

I only have the node dev environment on my end. Then I commit my changes to Github. Railway auto deploys


2 years ago

you do not have a Dockerfile?


drawingcontext
HOBBYOP

2 years ago

Nope


drawingcontext
HOBBYOP

2 years ago

Never had one for flask app to


drawingcontext
HOBBYOP

2 years ago

Didn’t know we need one


2 years ago

delete your Procfile


drawingcontext
HOBBYOP

2 years ago

ok


drawingcontext
HOBBYOP

2 years ago

deleted


drawingcontext
HOBBYOP

2 years ago

what about the start command in update config?


2 years ago

you should have a start script in your package.json


drawingcontext
HOBBYOP

2 years ago

"scripts": {

"start": "node app.js",

"dev": "app.js",

"test": "echo "Error: no test specified" && exit 1"

},


drawingcontext
HOBBYOP

2 years ago

yea, it has always been there


2 years ago

then you have no need for a procfile


drawingcontext
HOBBYOP

2 years ago

ok. but I am still getting the same problem


2 years ago

thats not valid syntax

1285420361617506405


drawingcontext
HOBBYOP

2 years ago

what should I enter in there?


drawingcontext
HOBBYOP

2 years ago

just node app.js?


drawingcontext
HOBBYOP

2 years ago

figured it out. TY for your help!


2 years ago

no problem!


drawingcontext
HOBBYOP

2 years ago

when you upgrade the plan, does the app needs to be redeloyed to take advantage of the new RAM etc?


Welcome!

Sign in to your Railway account to join the conversation.

Loading...