Celery Tasks Not Executing in Django Project

yaqcodesTRIAL

6 months ago

Dear Railway Support Team,

I'm experiencing an issue with my Django project "hotel_demo" deployed on Railway. Specifically, two Celery tasks,

send_control_request() and schedule_sub_asset_expiry()

are meant to be executed after a payment is made within my application. not being executed after a payment is verified. This functionality works correctly in my local environment, but fails in the Railway deployment even after setting up a separate process per the explanation here. I'll provide all relevant details about my project setup and the issue below.

Project Setup:

  1. Project Name: hotel_demo

  2. Framework: Django

  3. Celery Integration: Yes

  4. Message Broker: Redis

  5. Railway Services:

    • hotel_demo (main Django app)

    • Postgres (postgreSQL database)

    • Redis

    • celery_worker (Celery worker process)

Relevant Code and Configuration:

  1. celery.py:

import os from celery
import Celery from django.conf
import settings os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'hotel_demo.settings')
app = Celery('hotel_demo', broker=settings.CELERY_BROKER_URL, backend=settings.CELERY_BROKER_URL) app.config_from_object('django.conf:settings', namespace='CELERY') app.autodiscover_tasks()

  1. hotel_demo/init.py:

from .celery import app as celery_app __all__ = ('celery_app',)

  1. Procfile:

web: gunicorn hotel_demo.wsgi --log-file=- --access-logfile=- --error-logfile=- --capture-output
worker: celery -A hotel_demo worker --loglevel=info
beat: celery -A hotel_demo beat --loglevel=info

  1. settings.py (relevant parts):

CELERY_TIMEZONE = "UTC"
CELERY_ENABLE_UTC = True
CELERY_TASK_TRACK_STARTED = True
CELERY_TASK_TIME_LIMIT = 30 * 60
CELERY_BROKER_URL = os.environ.get('REDIS_URL', 'redis://localhost:6379/0')
CELERY_RESULT_BACKEND = os.environ.get('REDIS_URL', 'redis://localhost:6379/0') CELERY_BROKER_CONNECTION_RETRY_ON_STARTUP = True CELERY_ACCEPT_CONTENT = ['application/json'] CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
INSTALLED_APPS = [ # ...
'django_celery_results',
'django_celery_beat',
# ... ]

Issue Description: After a payment is verified, Celery tasks should be executed to update the status of a hotel room and send control commands. This process works correctly in my local environment but fails in the Railway deployment.

Local Behavior (Working):

[18/Oct/2024 00:42:45] "POST /api/payment/init/ HTTP/1.1" 200 105
[18/Oct/2024 00:43:16] "GET /api/payment/verify?trxref=TRYKEY-1729208564513&reference=TRYKEY-1729208564513 HTTP/1.1" 301 0
[18/Oct/2024 00:43:17] "GET /api/payment/verify/?trxref=TRYKEY-1729208564513&reference=TRYKEY-1729208564513 HTTP/1.1" 200 569
[18/Oct/2024 00:43:17] "POST /api/assets/TAS-0002-002/control/003/ HTTP/1.1" 200 42
Received message on rooms/TAS-0002-002/003/access: unlock Received message on rooms/TAS-0002-002/003/access: unlock
[18/Oct/2024 00:44:17] "POST /api/assets/TAS-0002-002/control/003/ HTTP/1.1" 200 42
Received message on rooms/TAS-0002-002/003/access: lock Received message on rooms/TAS-0002-002/003/access: lock

Railway Behavior (Not Working):

100.64.0.3 - - [18/Oct/2024:11:50:03 +0000] "POST /api/payment/init/ HTTP/1.1" 200 105 "https://www.trykeyprotocol.com/"; "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" 100.64.0.5 - - [18/Oct/2024:11:50:31 +0000] "GET /api/payment/verify?trxref=TRYKEY-1729252202398&reference=TRYKEY-1729252202398 HTTP/1.1" 301 0 "https://checkout.paystack.com/1liwb6lfywjerzg"; "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" Transaction TRYKEY-1729252202398 updated successfully Updated total revenue for asset TAS-0001-005 Updated HotelRoom 206 status and timestamps. New expiry: 2024-10-18 11:51:32.638256+00:00 100.64.0.5 - - [18/Oct/2024:11:50:33 +0000] "GET /api/payment/verify/?trxref=TRYKEY-1729252202398&reference=TRYKEY-1729252202398 HTTP/1.1" 200 5695 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"

As you can see, in the Railway logs, the transaction is updated, and the hotel room status is changed, but the control commands (which should be Celery tasks) are not being sent or executed.

Questions and Requests:

  1. Are there any issues with the Celery worker service in my Railway deployment?

  2. Can you confirm that the Redis service is properly connected and accessible to both the main Django app and the Celery worker?

  3. Is there anything in my configuration that might be preventing Celery tasks from running in the Railway environment?

  4. Could you please check if there are any environment variables or permissions that need to be set for Celery to function correctly in Railway?

Please let me know if you need any additional information to assist in troubleshooting.

Thank you for your time and assistance.

Best regards,
Yaqcodes

7 Replies

6 months ago

We do not support Procfiles with multiple entrypoints.

That is Heroku specific behavior that would be generally poor practice on our platform.

Instead, you want 3 Railway services, one service for gunicorn, one service for celery, and another service for beat.

Each of the 3 services will deploy from the same repo and have mostly the same environment variables.

The difference between the 3 services is that each will have a different start command to start the applicable application set in its service settings.


yaqcodesTRIAL

6 months ago

Thank you for your help brody. So all I need to do is remove the Procfile from my GitHub repo and define the start command for each service separately?


6 months ago

Correct.


yaqcodesTRIAL

6 months ago

I deleted the Procfile from my repo and put the gunicorn start command in my django service and the worker start command in my celery_worker service and it still does not work


yaqcodesTRIAL

6 months ago

I deleted the Procfile from my repo and put the gunicorn start command in my django service and the worker start command in my celery_worker service and it still does not work

Same output from the logs too


yaqcodesTRIAL

6 months ago

I've gotten it working now.
Turns out I wasn't deploying the same project
to my celery worker and even after doing that I ran into a Permission error which I solved by setting concurrency=1 as seen here
Thanks


6 months ago

Awsome!