2 months ago
I'm trying to deploy the Django Celery worker on Railway.
The worker starts successfully, but just hangs with no process until it crashes.
3 Replies
2 months ago
what broker are they using? like redis, rabbitmq, etc? and is it running on railway or somewhere else? and how are they starting the worker? like what's in their procfile or railway config? is it something like celery -A myproject worker -l info or different?
2 months ago
also helpful if you can grab any logs from railway
2 months ago
Had the same issue. Perhaps you could drop here a log or something to see if your problem is exiting just like that or e.g OOM (out of memory problem). Both can happen easily, need the right config. Mine was like exiting without any problem, and the cause was that when redis restarted itself, sometimes, celery could not, and it crashed. Under the hood celery is using a messaging service Kombu, and it was a known issue that when redis restarts, celery crashes. Update Kombu, include in the requirements.tyt and see if its crashing again. Moreover you I suggest you to include exit reasons for your celery beat and worker. If you use docker you could add to docker config this using dump-init:
RUN apt-get update && apt-get install -y dumb-init
ENTRYPOINT ["/usr/bin/dumb-init", "--"]
CMD ["celery", "-A", "your_app", "worker", "--loglevel=info"]
For railway you can add start commands for celery worker and beat:
start_worker.sh:
#!/bin/bash
cleanup() {
exit_code=$?
echo "[WORKER EXIT] Process exited with code: $exit_code"
if [ $exit_code -eq 137 ]; then
echo "[WORKER EXIT] Exit code 137 = OOM killed (SIGKILL)"
elif [ $exit_code -eq 143 ]; then
echo "[WORKER EXIT] Exit code 143 = SIGTERM (graceful shutdown)"
elif [ $exit_code -eq 1 ]; then
echo "[WORKER EXIT] Exit code 1 = General error"
fi
}
trap cleanup EXIT
echo "[WORKER START] Starting Celery worker at $(date)"
exec celery -A your_app worker --loglevel=info
for beat:
start_beat.sh:
#!/bin/bash
cleanup() {
exit_code=$?
echo "[BEAT EXIT] Process exited with code: $exit_code"
if [ $exit_code -eq 137 ]; then
echo "[BEAT EXIT] Exit code 137 = OOM killed"
fi
}
trap cleanup EXIT
echo "[BEAT START] Starting Celery beat at $(date)"
exec celery -A your_app beat --loglevel=info
and then you make a start executable command:
chmod +x start_worker.sh
./start_worker.sh
the above are the same then jsing this what they suggested:
celery -A myapp worker --loglevel=info
But adding exit logic. Let me know