a month ago
the title says everything.
25 Replies
a month ago
Hey there! We've found the following might help you get unblocked faster:
If you find the answer from one of these, please let us know by solving the thread!
a month ago
No. Doesn’t help. This is a support ticket.
a month ago
Project ID: 715a154f-c769-440d-9dad-373a1d7efafe
a month ago
I don’t need a bounty on this. The last thing I need when my website is down due to a Railway issue are irrelevant replies. There are already other reports about this. Please fix it. Before anyone asks: no. I haven’t changed anything lately.
a month ago
are you connecting to it via private network? are you seeing latency/connection count, vCPU/mem?
did you attempt to redeploy redis through the latest commit to see if anything will change?
a month ago
Some more details since I've been asked about this thread on Twitter after posting there:
Redis URL I connect with: redis://default:XXX@redis.railway.internal:6379
Errors: "Timeout reading from socket"
I tried restarting, redeploying, nothing helps.
Nothing has been adjusted in my app in the last 7 days.
No errors in Redis deployment itself but obviously web server and workers filled with the above error.
a month ago
No change in memory or CPU for the Redis deployment
a month ago
Can you try using the public URL instead of the one that uses the private network? This will allow us to pinpoint the issue
mahmoud-railway
Can you try using the public URL instead of the one that uses the private network? This will allow us to pinpoint the issue
a month ago
[Reply edited after 20 minutes]
After changing the URL to REDIS_PUBLIC_URL the app at least loads for some requests, but is barely responsive. It's pretty much the same effect.
As a start, I hope this proves that the issue is in Railway's internal networking and not a misconfiguration. This is not a long term solution. I shouldn't have to use the public URL to be able to connect.
a month ago
Application is barely responsive though.
a month ago
Glad to hear that it's working. Glad to see that your app is working now.
> Application is barely responsive though.
We're not going to mark the thread as resolved and keep it open. We'll investigate further and share a follow up. Feel free to ping me if I'm gone too long
mahmoud-railway
Glad to hear that it's working. Glad to see that your app is working now. > Application is barely responsive though.We're not going to mark the thread as resolved and keep it open. We'll investigate further and share a follow up. Feel free to ping me if I'm gone too long
a month ago
The app is pretty much down again. I'll let support do their thing, hopefully in a timely manner, but perhaps the issue isn't about private networking after all. Maybe it's volumes? I don't know.
a month ago
A failing request ID: wHVoMq7ITVCkuoGb0-QtfA
a month ago
(Python) Logs now show: redis.exceptions.ConnectionError: Connection closed by server.
brody
Can you provide more verbose logging? connection to what is being closed?
a month ago
Every component that tries to connect to Redis is timing out, the error is from redis.exceptions.ConnectionError , so connection to Redis is being closed. Do you want the stack traces?
Web server, which is responding with 499 to 99% of requests:
2025-10-31 15:28:19,345 root INFO serving RSS feed from DB..
[2025-10-31 15:28:20 +0200] [17] [INFO] connection closed
[2025-10-31 15:28:30 +0200] [17] [INFO] connection closed
[2025-10-31 15:28:40 +0200] [16] [INFO] connection closed
[2025-10-31 15:28:45 +0200] [16] [INFO] connection closed
[2025-10-31 15:29:00 +0200] [17] [INFO] connection closed
[2025-10-31 15:29:50 +0200] [16] [INFO] connection closed
[2025-10-31 15:30:00 +0200] [17] [INFO] connection closed
[2025-10-31 15:30:11 +0200] [17] [INFO] connection closed
[2025-10-31 15:30:42 +0200] [17] [INFO] connection closed
[2025-10-31 15:30:52 +0200] [15] [INFO] connection closed
[2025-10-31 15:31:02 +0200] [16] [INFO] connection closed
[2025-10-31 15:31:32 +0200] [17] [INFO] connection closed
[2025-10-31 15:31:42 +0200] [16] [INFO] connection closedWorkers:
File "/home/python/.local/lib/python3.12/site-packages/kombu/asynchronous/hub.py", line 373, in create_loop
cb(*cbargs)
File "/home/python/.local/lib/python3.12/site-packages/kombu/transport/redis.py", line 1352, in on_readable
self.cycle.on_readable(fileno)
File "/home/python/.local/lib/python3.12/site-packages/kombu/transport/redis.py", line 569, in on_readable
chan.handlers[type]()
File "/home/python/.local/lib/python3.12/site-packages/kombu/transport/redis.py", line 979, in _brpop_read
self.connection._deliver(loads(bytes_to_str(item)), dest)
[2025-10-31 15:23:38,993: WARNING/MainProcess] ???[???]: consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
File "/home/python/.local/lib/python3.12/site-packages/celery/worker/consumer/consumer.py", line 340, in start
blueprint.start(self)
File "/home/python/.local/lib/python3.12/site-packages/celery/bootsteps.py", line 116, in start
step.start(parent)
File "/home/python/.local/lib/python3.12/site-packages/celery/worker/consumer/consumer.py", line 746, in start
c.loop(*c.loop_args())
File "/home/python/.local/lib/python3.12/site-packages/celery/worker/loops.py", line 97, in asynloop
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/python/.local/lib/python3.12/site-packages/redis/client.py", line 3904, in _execute_transaction
self.parse_response(connection, '_')
File "/home/python/.local/lib/python3.12/site-packages/redis/client.py", line 3977, in parse_response
result = Redis.parse_response(
^^^^^^^^^^^^^^^^^^^^^
File "/home/python/.local/lib/python3.12/site-packages/redis/client.py", line 915, in parse_response
File "/home/python/.local/lib/python3.12/site-packages/kombu/transport/virtual/base.py", line 1017, in _deliver
callback(message)
File "/home/python/.local/lib/python3.12/site-packages/kombu/transport/virtual/base.py", line 638, in _callback
self.qos.append(message, message.delivery_tag)
File "/home/python/.local/lib/python3.12/site-packages/kombu/transport/redis.py", line 370, in append
.execute()
^^^^^^^^^
File "/home/python/.local/lib/python3.12/site-packages/sentry_sdk/integrations/redis/_sync_common.py", line 54, in sentry_patched_execute
return old_execute(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/python/.local/lib/python3.12/site-packages/redis/client.py", line 4019, in execute
return execute(conn, stack, raise_on_error)
response = connection.read_response()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/python/.local/lib/python3.12/site-packages/redis/connection.py", line 739, in read_response
response = self._parser.read_response()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/python/.local/lib/python3.12/site-packages/redis/connection.py", line 470, in read_response
self.read_from_socket()
File "/home/python/.local/lib/python3.12/site-packages/redis/connection.py", line 429, in read_from_socket
raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server.a month ago
Hello!
We've escalated your issue to our engineering team.
We aim to provide an update within 1 business day.
Please reply to this thread if you have any questions!
Status changed to Awaiting User Response Railway • about 1 month ago
a month ago
Hey shxkm I looked over your issue and have paged the relevant teams. We're working towards a fix!
Thank you so much for bringing this to our attention.
noahd
We have identified the issue and are working towards a fix.
a month ago
Thank you. This is a relief. As an engineer myself I'm kinda curious. If I had to bet I'd say it's volumes / noisy neighbour.
Status changed to Awaiting Railway Response Railway • about 1 month ago
shxkm
Thank you. This is a relief. As an engineer myself I'm kinda curious. If I had to bet I'd say it's volumes / noisy neighbour.
a month ago
You can monitor it via https://status.railway.com/cmhdlky9r003e44y4p4lyortz
We'll post updates there as it comes.
This is not something which should be happening and we are taking steps to prevent it from reoccurring in the future. Thank you for your patience 
Status changed to Awaiting User Response Railway • about 1 month ago
a month ago
We have implemented a fix. Things should be coming back online here shortly.
noahd
We have implemented a fix. Things should be coming back online here shortly.
a month ago
Does that mean I can switch back to the internal DNS again?
Status changed to Awaiting Railway Response Railway • about 1 month ago
shxkm
Does that mean I can switch back to the internal DNS again?
a month ago
Can you try and let me know how it goes?
Status changed to Awaiting User Response Railway • about 1 month ago
noahd
Can you try and let me know how it goes?
a month ago
Yeah this is the longest the app has been responsive for a while. Thanks.
Status changed to Awaiting Railway Response Railway • about 1 month ago
Status changed to Solved noahd • about 1 month ago
a month ago
✅ The ticket Service Timeout Issue has been marked as completed.