container crashed
drawingcontext
HOBBYOP

2 years ago

container worked and everything was working well but container crashed…not sure why..Here's the projectid - 1b883543-ff8a-4389-9f72-1fe4b7b73668

161 Replies

2 years ago

Is there anything in the logs that might show why it crashed?


drawingcontext
HOBBYOP

2 years ago

/bin/bash: line 1: web:: command not found

container event container restart

/bin/bash: line 1: web:: command not found

container event container died


drawingcontext
HOBBYOP

2 years ago

Also when the app was up, it timedout on few api calls and I tried web: gunicorn app:app --timeout 120


drawingcontext
HOBBYOP

2 years ago

but I am back to web: gunicorn app:app in procfile


2 years ago

web: is a Procfile thing, you should be able to set your start command to just gunicorn app:web --timeout 120

Railway hasn't supported procfiles in a long time


drawingcontext
HOBBYOP

2 years ago

File "/opt/venv/lib/python3.11/site-packages/gunicorn/util.py", line 402, in import_app

app = getattr(mod, name)

      ^^^^^^^^^^^^^^^^^^

AttributeError: module 'app' has no attribute 'web'

Failed to find attribute 'web' in 'app'.

[2024-09-13 14:09:51 +0000] [4] [INFO] Worker exiting (pid: 4)

[2024-09-13 14:09:51 +0000] [1] [INFO] Shutting down: Master

[2024-09-13 14:09:51 +0000] [1] [INFO] Reason: App failed to load.

container event container restart

[2024-09-13 14:09:52 +0000] [1] [INFO] Starting gunicorn 20.1.0

[2024-09-13 14:09:52 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)

[2024-09-13 14:09:52 +0000] [1] [INFO] Using worker: sync

[2024-09-13 14:09:52 +0000] [4] [INFO] Booting worker with pid: 4

Traceback (most recent call last):

File "/opt/venv/lib/python3.11/site-packages/gunicorn/util.py", line 402, in import_app

app = getattr(mod, name)

      ^^^^^^^^^^^^^^^^^^

AttributeError: module 'app' has no attribute 'web'

Failed to find attribute 'web' in 'app'.

[2024-09-13 14:09:52 +0000] [4] [INFO] Worker exiting (pid: 4)

[2024-09-13 14:09:52 +0000] [1] [INFO] Shutting down: Master

[2024-09-13 14:09:52 +0000] [1] [INFO] Reason: App failed to load.

container event container died

[2024-09-13 14:09:53 +0000] [1] [INFO] Starting gunicorn 20.1.0

[2024-09-13 14:09:53 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)

[2024-09-13 14:09:53 +0000] [1] [INFO] Using worker: sync

[2024-09-13 14:09:53 +0000] [4] [INFO] Booting worker with pid: 4

Traceback (most recent call last):

File "/opt/venv/lib/python3.11/site-packages/gunicorn/util.py", line 402, in import_app

app = getattr(mod, name)

      ^^^^^^^^^^^^^^^^^^

AttributeError: module 'app' has no attribute 'web'

Failed to find attribute 'web' in 'app'.

[2024-09-13 14:09:53 +0000] [4] [INFO] Worker exiting (pid: 4)

container event container restart

[2024-09-13 14:09:54 +0000] [1] [INFO] Shutting down: Master

[2024-09-13 14:09:54 +0000] [1] [INFO] Reason: App failed to load.

[2024-09-13 14:09:55 +0000] [1] [INFO] Starting gunicorn 20.1.0

[2024-09-13 14:09:55 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)

[2024-09-13 14:09:55 +0000] [1] [INFO] Using worker: sync

[2024-09-13 14:09:55 +0000] [4] [INFO] Booting worker with pid: 4

Traceback (most recent call last):

File "/opt/venv/lib/python3.11/site-packages/gunicorn/util.py", line 402, in import_app

app = getattr(mod, name)

      ^^^^^^^^^^^^^^^^^^

AttributeError: module 'app' has no attribute 'web'

Failed to find attribute 'web' in 'app'.

[2024-09-13 14:09:55 +0000] [4] [INFO] Worker exiting (pid: 4)

[2024-09-13 14:09:55 +0000] [1] [INFO] Shutting down: Master

[2024-09-13 14:09:55 +0000] [1] [INFO] Reason: App failed to load.

container event container died

[2024-09-13 14:09:56 +0000] [1] [INFO] Starting gunicorn 20.1.0

[2024-09-13 14:09:56 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)

[2024-09-13 14:09:56 +0000] [1] [INFO] Using worker: sync

[2024-09-13 14:09:56 +0000] [4] [INFO] Booting worker with pid: 4

Traceback (most recent call last):

File "/opt/venv/lib/python3.11/site-packages/gunicorn/util.py", line 402, in import_app

app = getattr(mod, name)

      ^^^^^^^^^^^^^^^^^^

AttributeError: module 'app' has no attribute 'web'

Failed to find attribute 'web' in 'app'.

[2024-09-13 14:09:56 +0000] [4] [INFO] Worker exiting (pid: 4)

[2024-09-13 14:09:56 +0000] [1] [INFO] Shutting down: Master

[2024-09-13 14:09:56 +0000] [1] [INFO] Reason: App failed to load.

container event container died

[2024-09-13 14:09:57 +0000] [1] [INFO] Using worker: sync


2 years ago

Can you share your repo? Your application might be called app and not web, in which case, you would use app:app


2 years ago

yes we do


drawingcontext
HOBBYOP

2 years ago

That is true.


drawingcontext
HOBBYOP

2 years ago

I am restarting with app:app


drawingcontext
HOBBYOP

2 years ago

Now the app is up. testing the api calls to see if it times out


drawingcontext
HOBBYOP

2 years ago

Same issue


drawingcontext
HOBBYOP

2 years ago

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): feed-api.cielo.finance:443

[2024-09-13 14<:16:1199656381486219374>51 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:4)

[2024-09-13 14<:16:1199656381486219374>51 +0000] [4] [INFO] Worker exiting (pid: 4)

[2024-09-13 14<:16:1199656381486219374>51 +0000] [5] [INFO] Booting worker with pid: 5


drawingcontext
HOBBYOP

2 years ago

It is a worker timeout


drawingcontext
HOBBYOP

2 years ago

Check the update


2 years ago

how long do you expect a single request to take


drawingcontext
HOBBYOP

2 years ago

it depends on the response but should not take more than 5-7 seconds tbh


2 years ago

when testing, how many RPS are you doing?


drawingcontext
HOBBYOP

2 years ago

what is RPS?


drawingcontext
HOBBYOP

2 years ago

locally it works


2 years ago

requests per second


drawingcontext
HOBBYOP

2 years ago

It is adata analysis application hitting many different apis


drawingcontext
HOBBYOP

2 years ago

so def. 100 req/sec


2 years ago

then you are likely running into issues with having run gunicorn with sync workers


2 years ago

start it with the uvicorn event worker


drawingcontext
HOBBYOP

2 years ago

how do I do it?


2 years ago

what's your current start command


drawingcontext
HOBBYOP

2 years ago

gunicorn app:app --timeout 120


drawingcontext
HOBBYOP

2 years ago

gunicorn app:app --timeout 120


2 years ago

no need to ping reply


2 years ago

what kind of app? fastapi, flask?


drawingcontext
HOBBYOP

2 years ago

flask


2 years ago

your start command would become gunicorn app:app --timeout 120 --worker-class eventlet
you will also need to make sure eventlet is in your requirements.txt file


drawingcontext
HOBBYOP

2 years ago

Here's my current req


drawingcontext
HOBBYOP

2 years ago

Flask==2.0.3
Werkzeug==2.0.3
Jinja2==3.0.3
itsdangerous==2.0.1
click==8.0.4
gunicorn==20.1.0
requests==2.26.0
prettytable==2.5.0
Flask-Login==0.5.0


drawingcontext
HOBBYOP

2 years ago

what version should I use for eventlet


2 years ago

the latest is probably fine


drawingcontext
HOBBYOP

2 years ago

ok, made changes and redeploying


drawingcontext
HOBBYOP

2 years ago

Error: class uri 'eventlet' invalid or not found:

[Traceback (most recent call last):

File "/opt/venv/lib/python3.11/site-packages/gunicorn/util.py", line 99, in load_class

mod = importlib.import_module('.'.join(components))

      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/root/.nix-profile/lib/python3.11/importlib/init.py", line 126, in import_module

return _bootstrap._gcd_import(name[level:], package, level)

       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "", line 1204, in gcdimport

File "", line 1176, in findand_load

File "", line 1147, in findandloadunlocked

File "", line 690, in loadunlocked

File "", line 940, in exec_module

File "", line 241, in callwithframesremoved

File "/opt/venv/lib/python3.11/site-packages/gunicorn/workers/geventlet.py", line 20, in

from eventlet.wsgi import ALREADY_HANDLED as EVENTLET_ALREADY_HANDLED

ImportError: cannot import name 'ALREADY_HANDLED' from 'eventlet.wsgi' (/opt/venv/lib/python3.11/site-packages/eventlet/wsgi.py)

]

container event container died


2 years ago

do you have it in your requirements.txt?


drawingcontext
HOBBYOP

2 years ago

Flask==2.0.3
Werkzeug==2.0.3
Jinja2==3.0.3
itsdangerous==2.0.1
click==8.0.4
gunicorn==20.1.0
requests==2.26.0
prettytable==2.5.0
Flask-Login==0.5.0
eventlet==0.33.3


2 years ago

whats your start command


drawingcontext
HOBBYOP

2 years ago

shoujd I use gevent?


2 years ago

^


drawingcontext
HOBBYOP

2 years ago

gunicorn app:app --timeout 120 --worker-class eventlet


drawingcontext
HOBBYOP

2 years ago

I have a Procfile


drawingcontext
HOBBYOP

2 years ago

does it matter for Railway


2 years ago

no, we support procfiles


2 years ago

okay try gevent


drawingcontext
HOBBYOP

2 years ago

trying now


drawingcontext
HOBBYOP

2 years ago

invalid type: string "gunicorn --timeout 120 --worker-class gevent app:app", expected a ma


drawingcontext
HOBBYOP

2 years ago

did I get format right?


2 years ago

try moving app:app to after gunicorn


drawingcontext
HOBBYOP

2 years ago

context: 5f53c0eeaf434d6d998e9ed8ac0a2c0f

Nixpacks build failed

Error: Reading Procfile

Caused by:

invalid type: string "gunicorn app:app --timeout 120 --worker-class gevent", expected a map


drawingcontext
HOBBYOP

2 years ago

just way beyond my understanding…


drawingcontext
HOBBYOP

2 years ago

gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -I/nix/store/wwg6376ymkhpjxy6c3w9gkbhv6l2gmqy-libxcrypt-4.4.36/include -fPIC -DLIBEVEMBED=1 -DEVCOMMON= -DEVCLEANUPENABLE=0 -DEVEMBEDENABLE=0 -DEVPERIODICENABLE=0 -DEVUSEREALTIME=1 -DEVUSEMONOTONIC=1 -DEVUSEFLOOR=1 -Isrc/gevent/libev -I/nix/store/k3701zl6gmx3la7y4dnflcvf3xfy88kh-python3-3.11.9/include/python3.11 -I/root/.nix-profile/include/python3.11 -I/tmp/pip-install-mht4g7p5/gevent7b95d12f6294466681fc5ce8f6126ea3/deps -I/tmp/pip-install-mht4g7p5/gevent7b95d12f6294466681fc5ce8f6126ea3/src/gevent/libev -I/tmp/pip-install-mht4g7p5/gevent7b95d12f6294466681fc5ce8f6126ea3/deps/libev -Isrc/gevent -Isrc/gevent/libev -Isrc/gevent/resolver -I. -I/opt/venv/include -I/nix/store/k3701zl6gmx3la7y4dnflcvf3xfy88kh-python3-3.11.9/include/python3.11 -c src/gevent/libev/callbacks.c -o build/temp.linux-x8664-cpython-311/src/gevent/libev/callbacks.o

11 103.8 gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -I/nix/store/wwg6376ymkhpjxy6c3w9gkbhv6l2gmqy-libxcrypt-4.4.36/include -fPIC -DLIBEVEMBED=1 -DEVCOMMON= -DEVCLEANUPENABLE=0 -DEVEMBEDENABLE=0 -DEVPERIODICENABLE=0 -DEVUSEREALTIME=1 -DEVUSEMONOTONIC=1 -DEVUSEFLOOR=1 -Isrc/gevent/libev -I/nix/store/k3701zl6gmx3la7y4dnflcvf3xfy88kh-python3-3.11.9/include/python3.11 -I/root/.nix-profile/include/python3.11 -I/tmp/pip-install-mht4g7p5/gevent7b95d12f6294466681fc5ce8f6126ea3/deps -I/tmp/pip-install-mht4g7p5/gevent7b95d12f6294466681fc5ce8f6126ea3/src/gevent/libev -I/tmp/pip-install-mht4g7p5/gevent7b95d12f6294466681fc5ce8f6126ea3/deps/libev -Isrc/gevent -Isrc/gevent/libev -Isrc/gevent/resolver -I. -I/opt/venv/include -I/nix/store/k3701zl6gmx3la7y4dnflcvf3xfy88kh-python3-3.11.9/include/python3.11 -c src/gevent/libev/corecext.c -o build/temp.linux-x8664-cpython-311/src/gevent/libev/corecext.o

11 103.8 In file included from src/gevent/libev/libev.h:9,

11 103.8 from src/gevent/libev/corecext.c:1319:

11 103.8 /tmp/pip-install-mht4g7p5/gevent_7b95d12f6294466681fc5ce8f6126ea3/deps/libev/ev.c:580:48: warning: "/*" within comment [-Wcomment]

11 103.8 580 | /#define MIN_INTERVAL 0.00000095367431640625 / 1/2**20, good till 2200 */

11 103.8 |

11 103.8 /tmp/pip-install-mht4g7p5/gevent7b95d12f6294466681fc5ce8f6126ea3/deps/libev/ev.c: In function ‘ecbbinary32tobinary16’:


drawingcontext
HOBBYOP

2 years ago

gone back to eventlet


drawingcontext
HOBBYOP

2 years ago

File "/opt/venv/lib/python3.11/site-packages/eventlet/greenio/base.py", line 32, in

socket_timeout = eventlet.timeout.wrap_is_timeout(socket.timeout)

                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/opt/venv/lib/python3.11/site-packages/eventlet/timeout.py", line 166, in wrapistimeout

[base.is](base.is)_timeout = property(lambda _: True)

^^^^^^^^^^^^^^^

TypeError: cannot set 'is_timeout' attribute of immutable type 'TimeoutError'

]

container event container died


drawingcontext
HOBBYOP

2 years ago

can I use python-3.9.7


2 years ago

yes, set it in a runtime.txt file


drawingcontext
HOBBYOP

2 years ago

I did that but no idea how to resolve the issue with timeouts


drawingcontext
HOBBYOP

2 years ago

neither gevent or eventlet is working on railway


2 years ago

is your app now using gevent


drawingcontext
HOBBYOP

2 years ago

gevent had build issues so I went back to eventlet


drawingcontext
HOBBYOP

2 years ago

but eventlet is crashing the docker


drawingcontext
HOBBYOP

2 years ago

Error: class uri 'eventlet' invalid or not found:


2 years ago

you woud need to get one of them working


drawingcontext
HOBBYOP

2 years ago

runtime.txt does not seem to be getting picked up


2 years ago

why do you say that


drawingcontext
HOBBYOP

2 years ago

File "/opt/venv/lib/python3.11


drawingcontext
HOBBYOP

2 years ago

using python3.11


2 years ago

i cant verify that, you are about to delete the service


drawingcontext
HOBBYOP

2 years ago

I deleted the service


drawingcontext
HOBBYOP

2 years ago

started a new one.


drawingcontext
HOBBYOP

2 years ago

and I got things back to a place where it is working


drawingcontext
HOBBYOP

2 years ago

not using eventlet or gevent


drawingcontext
HOBBYOP

2 years ago

just timeout


drawingcontext
HOBBYOP

2 years ago

I want to see if the error persists


drawingcontext
HOBBYOP

2 years ago

atleast now I don't have build issues or deployment issues


2 years ago

this wasnt an issue with railway so the timeout issue isnt going to change


drawingcontext
HOBBYOP

2 years ago

no timeouts locally


drawingcontext
HOBBYOP

2 years ago

local dev works fine


2 years ago

you are not running sync workers locally


2 years ago

are you sure this is flask?


drawingcontext
HOBBYOP

2 years ago

Yes


drawingcontext
HOBBYOP

2 years ago

app = Flask(name)


drawingcontext
HOBBYOP

2 years ago

same error


drawingcontext
HOBBYOP

2 years ago

[2024-09-13 16<:16:1199656381486219374>28 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:4)

[2024-09-13 16<:16:1199656381486219374>28 +0000] [4] [INFO] Worker exiting (pid: 4)

[2024-09-13 16<:16:1199656381486219374>28 +0000] [5] [INFO] Booting worker with pid: 5


2 years ago

you need to use an async worker


drawingcontext
HOBBYOP

2 years ago

that will require changes ot my code?


drawingcontext
HOBBYOP

2 years ago

I just want to replicate my local environment on railway


2 years ago

im not sure


2 years ago

in a production environment you must use gunicorn


drawingcontext
HOBBYOP

2 years ago

yes that works


2 years ago

and in your case you also need to use async workers


drawingcontext
HOBBYOP

2 years ago

but the moment I add eventlet, it is not even building


drawingcontext
HOBBYOP

2 years ago

the moment I add gevent, docker is crashing


2 years ago

try gthread then


drawingcontext
HOBBYOP

2 years ago

increasaed timeout sufficiently large so that works but towards the end, the final request the response got chopped off


drawingcontext
HOBBYOP

2 years ago

is there a limit on the size of response for railway?


2 years ago

no, only a 5 minute time limit


drawingcontext
HOBBYOP

2 years ago

this was the last step before rendering if you see logs


2 years ago

increasing the time-out here is just a Band-Aid solution


drawingcontext
HOBBYOP

2 years ago

still dont kknow why railway has issue.


drawingcontext
HOBBYOP

2 years ago

works fine on local


drawingcontext
HOBBYOP

2 years ago

works well on a vps server


drawingcontext
HOBBYOP

2 years ago

fails on railway


2 years ago

that not not indicate an issue with railway, that indicates an issue with your configurations on railway


2 years ago

you need to use an async worker so that your app can handle more then 3 requests concurrently


drawingcontext
HOBBYOP

2 years ago

yea, will need to implement celery and redis but was hoping that I can do that later.


drawingcontext
HOBBYOP

2 years ago

Maybe for now, I just use VPS and evaluate railway later on


2 years ago

you can do celery and redis later, right now you need to use an async worker


drawingcontext
HOBBYOP

2 years ago

didn't we try that?


drawingcontext
HOBBYOP

2 years ago

eventlet and gevent both crash the docker for whatever reason


2 years ago

you have something misconfigured and will need to look into why that's happening


drawingcontext
HOBBYOP

2 years ago

I'll modify the code to see if it works


drawingcontext
HOBBYOP

2 years ago

by importing asyncio & aiohttp and adding async with in front of all my request fetch functions


drawingcontext
HOBBYOP

2 years ago

Figured out a way.


drawingcontext
HOBBYOP

2 years ago

Is there a way to hard refresh a web page?


drawingcontext
HOBBYOP

2 years ago

feels like it is serving some cached version


2 years ago

railway doesnt do any asset caching, is it behind cloudflare?


drawingcontext
HOBBYOP

a year ago

I migrated the Flask App to NodeJS app


drawingcontext
HOBBYOP

a year ago

is the startup command - web: node app.js


drawingcontext
HOBBYOP

a year ago

container is failing right now


drawingcontext
HOBBYOP

a year ago

Build is successful


a year ago

please provide the error


drawingcontext
HOBBYOP

a year ago

/bin/bash: line 1: web:: command not found

container event container died


a year ago

what is the contents of your procfile


drawingcontext
HOBBYOP

a year ago

web: node app.js


a year ago

I'll check back with this when I'm back at the computer so I can view your logs, I don't think you are sending me the correct error


drawingcontext
HOBBYOP

a year ago

\

1285242638093779000


a year ago

wait, aren't you deploying from a Dockerfile? why do you have a procfile


drawingcontext
HOBBYOP

a year ago

I connected my github project from railway


drawingcontext
HOBBYOP

a year ago

railway handles everything


drawingcontext
HOBBYOP

a year ago

I can remove the procfile


drawingcontext
HOBBYOP

a year ago

you want me to try that?


a year ago

haha slow down there


drawingcontext
HOBBYOP

a year ago

ok, let me know


drawingcontext
HOBBYOP

a year ago

trying my luck …


a year ago

send me your Dockerfile please


drawingcontext
HOBBYOP

a year ago

How do I do it?


drawingcontext
HOBBYOP

a year ago

I only have the node dev environment on my end. Then I commit my changes to Github. Railway auto deploys


a year ago

you do not have a Dockerfile?


drawingcontext
HOBBYOP

a year ago

Nope


drawingcontext
HOBBYOP

a year ago

Never had one for flask app to


drawingcontext
HOBBYOP

a year ago

Didn’t know we need one


a year ago

delete your Procfile


drawingcontext
HOBBYOP

a year ago

ok


drawingcontext
HOBBYOP

a year ago

deleted


drawingcontext
HOBBYOP

a year ago

what about the start command in update config?


a year ago

you should have a start script in your package.json


drawingcontext
HOBBYOP

a year ago

"scripts": {
"start": "node app.js",
"dev": "app.js",
"test": "echo \"Error: no test specified\" && exit 1"
},


drawingcontext
HOBBYOP

a year ago

yea, it has always been there


a year ago

then you have no need for a procfile


drawingcontext
HOBBYOP

a year ago

ok. but I am still getting the same problem


a year ago

thats not valid syntax

1285420361617506300


drawingcontext
HOBBYOP

a year ago

what should I enter in there?


drawingcontext
HOBBYOP

a year ago

just node app.js?


drawingcontext
HOBBYOP

a year ago

figured it out. TY for your help!


a year ago

no problem!


drawingcontext
HOBBYOP

a year ago

when you upgrade the plan, does the app needs to be redeloyed to take advantage of the new RAM etc?


Loading...