Celery connection issue
Anonymous
TRIALOP

2 years ago

Hi! Just recently I've been having issues with my celery instance. It was working all this time. It can't connect to redis (everything is dockerized). Here is the following issue: Cannot connect to {redacted redis address} Lookup timed out.. - Has anyone had the same issue?

102 Replies

Anonymous
TRIALOP

2 years ago

N/A


brody
EMPLOYEE

2 years ago

the redis address would be very important to help us debug this



Anonymous
TRIALOP

2 years ago

🙂


brody
EMPLOYEE

2 years ago

oh so not on railway


brody
EMPLOYEE

2 years ago

is there some kind of whitelist you need to do?


Anonymous
TRIALOP

2 years ago

well I also tried railway as well


Anonymous
TRIALOP

2 years ago

I'm having the same issue no matter the redis address


Anonymous
TRIALOP

2 years ago

however it works just fine when connecting locally


brody
EMPLOYEE

2 years ago

can we focus on getting you connected to a railway hosted redis database?


Anonymous
TRIALOP

2 years ago

local instance to production redis


Anonymous
TRIALOP

2 years ago

that's dine with me


Anonymous
TRIALOP

2 years ago

fine*


brody
EMPLOYEE

2 years ago

works better for me since i know nothing about redislabs


Anonymous
TRIALOP

2 years ago

monorail.proxy.rlwy.net:34378


brody
EMPLOYEE

2 years ago

can i see a screenshot of your railway project please?


Anonymous
TRIALOP

2 years ago

sure


Anonymous
TRIALOP

2 years ago

give me a sec, let me configure redis on railway


brody
EMPLOYEE

2 years ago

if redis isn't configured, where did you get that domain from?


Anonymous
TRIALOP

2 years ago

so that domain is from a heroku instance, I'm having the same issue and decided to try railway to see if it was a isolated instance


Anonymous
TRIALOP

2 years ago

My application had been working just fine up until today


brody
EMPLOYEE

2 years ago

im talking about this domain


Anonymous
TRIALOP

2 years ago

Oh I had tested a setup


Anonymous
TRIALOP

2 years ago

earlier


Anonymous
TRIALOP

2 years ago

I just tried connecting to it and i do get the same error


Anonymous
TRIALOP

2 years ago

[2024-02-26 17:31:11,560: ERROR/MainProcess] consumer: Cannot connect to redis://default:**@[monorail.proxy.rlwy.net:34378](monorail.proxy.rlwy.net:34378)/0: Error -3 connecting to monorail.proxy.rlwy.net:34378. Lookup timed out..


brody
EMPLOYEE

2 years ago

please show me the screenshot of your project


Anonymous
TRIALOP

2 years ago

just a normal screenshot? or is there a railway function


Anonymous
TRIALOP

2 years ago

sorry, new to this


Anonymous
TRIALOP

2 years ago


Anonymous
TRIALOP

2 years ago

ignore the flask


brody
EMPLOYEE

2 years ago

okay can show me how you are trying to connect to it


Anonymous
TRIALOP

2 years ago

let me make a quick loom



Anonymous
TRIALOP

2 years ago

Thanks for the help!


Anonymous
TRIALOP

2 years ago

I truly appreciate it


brody
EMPLOYEE

2 years ago

very helpful video, what do you have REDIS_ENDPOINT and REDIS_PORT set to in your service variables?


Anonymous
TRIALOP

2 years ago

give me one sec


Anonymous
TRIALOP

2 years ago

REDISENDPOINT=default:**@monorail.proxy.rlwy.net REDISPORT=34378


Anonymous
TRIALOP

2 years ago

asterisk is the password btw


brody
EMPLOYEE

2 years ago

instead of trying to build the url yourself, just use a REDIS_URL variable in the celery config file, then in your service variables set REDIS_URL to ${{Redis.REDIS_PRIVATE_URL}}
this is unlikely to fix the main issue, but its best we do this instead


Anonymous
TRIALOP

2 years ago

okay got it - would my config look like this then?

import os
from dotenv import loaddotenv loaddotenv()

REDISURL = os.environ.get('REDISURL', '6379') # default to 6379 if not provided
imports = ["tasks"]

brokerurl = REDISURL
resultbackend = REDISURL
taskserializer = 'json' resultserializer = 'json'
acceptcontent = ['json'] timezone = 'America/Chicago' enableutc = True
brokerconnectionretryonstartup = True


brody
EMPLOYEE

2 years ago

lets omit the dotenv for now, since it looks like you are committing that file to your repo


Anonymous
TRIALOP

2 years ago

okay gotcha


brody
EMPLOYEE

2 years ago

import os

imports = ["tasks"]

broker_url = os.environ['REDIS_URL']
result_backend = os.environ['REDIS_URL']
task_serializer = 'json'
result_serializer = 'json'
accept_content = ['json']
timezone = 'America/Chicago'
enable_utc = True
broker_connection_retry_on_startup = True

Anonymous
TRIALOP

2 years ago

oaky got it - railway gives me two options REDISURL and REDISPRIVATE_URL


Anonymous
TRIALOP

2 years ago

when I'm setting env varible


brody
EMPLOYEE

2 years ago

please reference my message


Anonymous
TRIALOP

2 years ago

okay got it, deploying now


Anonymous
TRIALOP

2 years ago

okay deployed -s ame issue


brody
EMPLOYEE

2 years ago

show me the new error please


Anonymous
TRIALOP

2 years ago

[2024-02-26 17:54:49,592: ERROR/MainProcess] consumer: Cannot connect to redis://default:**@redis.railway.internal:6379//: Error -3 connecting to redis.railway.internal:6379. Lookup timed out..


brody
EMPLOYEE

2 years ago

cool, send your dockerfile please, and going forward please enclose logs and code (or similar) in code blocks


Anonymous
TRIALOP

2 years ago

sorry haha , how do i enclose in code blocks


brody
EMPLOYEE

2 years ago

triple backticks


Anonymous
TRIALOP

2 years ago


brody
EMPLOYEE

2 years ago

try this instead

# Dockerfile.worker

# Use an official Python runtime as a parent image
FROM python:3.11.1

WORKDIR /app

COPY requirements.txt .
RUN pip install -r requirements.txt

COPY . .

# Command to run the Celery worker
CMD sleep 3 && celery -A tasks worker --loglevel=info --pool=eventlet

Anonymous
TRIALOP

2 years ago

trying


Anonymous
TRIALOP

2 years ago

same issue


brody
EMPLOYEE

2 years ago

interesting


Anonymous
TRIALOP

2 years ago

yeah, I'm not sure if it's a rediscloud issue but status have been okay with a few urgent fixes that happend last two days


Anonymous
TRIALOP

2 years ago

as I said before, everything worked properly until today


Anonymous
TRIALOP

2 years ago

does railway use the rediscloud service?


brody
EMPLOYEE

2 years ago

railway deploys a redis docker image


Anonymous
TRIALOP

2 years ago

hmm.


Anonymous
TRIALOP

2 years ago

yeah, it's really odd


Anonymous
TRIALOP

2 years ago

driving me insane


brody
EMPLOYEE

2 years ago

increase the sleep to 10 seconds?


Anonymous
TRIALOP

2 years ago

sure ill try it


Anonymous
TRIALOP

2 years ago

same issue :/


brody
EMPLOYEE

2 years ago

im kinda out of ideas, you sure you are usng the dockerfile i provided?


Anonymous
TRIALOP

2 years ago

Yes I am :9


Anonymous
TRIALOP

2 years ago

😦


Anonymous
TRIALOP

2 years ago

I'm dumbfounded as well. It literally worked for months before today


brody
EMPLOYEE

2 years ago

and you haven't modified anything at all about the redis database on railway right? absolutely nothing at all?


Anonymous
TRIALOP

2 years ago

Nope


Anonymous
TRIALOP

2 years ago

nothing


Anonymous
TRIALOP

2 years ago

I even tried rabbitmq instead of redis


Anonymous
TRIALOP

2 years ago

same issue


brody
EMPLOYEE

2 years ago

let me do some thinking, could you share your repo, or add me to it?


Anonymous
TRIALOP

2 years ago

Thank you! This is under a client repo but I can create another and add you in


brody
EMPLOYEE

2 years ago

yes please
brody192


Anonymous
TRIALOP

2 years ago

Hmm


Anonymous
TRIALOP

2 years ago

well, here's another thing


Anonymous
TRIALOP

2 years ago

I tried a skeleton celery


Anonymous
TRIALOP

2 years ago

and it's connecting


Anonymous
TRIALOP

2 years ago

I'm going to look into this further


Anonymous
TRIALOP

2 years ago

I'll keep you posted 🙂


Anonymous
TRIALOP

2 years ago

Thank you Brody


Anonymous
TRIALOP

2 years ago

I will definitley buy you some coffee


brody
EMPLOYEE

2 years ago

theres no need but i very much appreciate the gesture


Anonymous
TRIALOP

2 years ago

Here you go ! https://www.buymeacoffee.com/brody192/c/8635280 - The world needs mroe people like you. Much appreciated


Anonymous
TRIALOP

2 years ago

I figured out what it was - it was the concurrency pool. Eventlet doesn't want to play nice with Docker I guess


brody
EMPLOYEE

2 years ago

sorry i couldn’t solve this but happy you have! and thank you so much for the train!!


brody
EMPLOYEE

2 years ago

a sleep is still needed, but you can bump the sleep down to 3 seconds


Anonymous
TRIALOP

2 years ago

Okay sounds good! Why do you recommend the sleep?


Anonymous
TRIALOP

2 years ago

You still helped, I appreciate it


brody
EMPLOYEE

2 years ago

because the private network takes about 3 seconds to be able to respond to dns lookups


Anonymous
TRIALOP

2 years ago

Ah okay gotcha


Anonymous
TRIALOP

2 years ago

makes sense


Anonymous
TRIALOP

2 years ago

thank you again


brody
EMPLOYEE

2 years ago

happy to help where i can!


Loading...