a month ago
Summary
Our Redis service restarted unexpectedly today. After it came back up, our
long-running application services can no longer talk to it: Redis commands
time out and our app health check reports Redis as "unhealthy". The Redis
instance itself looks healthy in the Railway metrics panel, so this appears
to be a connectivity problem between our app services and the Redis service
after the restart.
Timeline (UTC)
-
2026-07-18 ~17:15 UTC: errors started across all our services.
Initial symptom: "connect ETIMEDOUT" to Redis (socket could not open).
-
Redis observability panel then showed Uptime = 2m, i.e. the Redis service
had just restarted. Data was restored from RDB (~590K keys), memory ~99MB,
Slow Log 0, 39 connected clients, eviction policy noeviction, maxmemory
unlimited. So the Redis server itself recovered and looks healthy.
-
After the restart the symptom changed to "Command timed out" (socket opens
but commands never get a reply), which is still ongoing at a steady rate.
Current state
-
Redis service: healthy per Railway metrics (uptime a few minutes, clients
connected, RDB ok, slow log 0).
-
Our app services (Api, workers) have NOT restarted (process uptime 2d 12h)
and are stuck: their Redis client pools time out on every command.
-
Our app health endpoint reports Redis = unhealthy and cannot read
Redis INFO (Memory = N/A, Keys = N/A), while Database (Postgres) is healthy
with 5ms latency.
-
Error rate is steady at ~500-700 errors/min (down from a ~2500/min peak but
not recovering on its own).
Representative error messages
-
"Failed to check blacklist: Command timed out"
-
"Failed to check blacklist: Reached the max retries per request limit
(which is 2)"
-
"Redis connection error: connect ETIMEDOUT"
-
"Redis connection error: read ETIMEDOUT" / "read ECONNRESET"
-
"Flush buffers error: Command timed out"
-
"Redis idempotency error" / "Idempotency store unavailable"
-
Client library reconnect backoff climbed to attempt #26 (~13s interval).
Questions
-
Why did the Redis service restart around 17:15 UTC? Was there a platform
event, OOM, or maintenance/failover on our Redis plugin?
-
Did the Redis service's internal/private networking endpoint (host/IP)
change on restart? We suspect our long-lived clients are holding stale
connections to the previous instance.
-
Is there anything on the platform side preventing new connections/commands
from our app services to the Redis service right now?
We are restarting our app services to force fresh Redis connections, but we
want to understand the root cause of the Redis restart and whether the
private-network endpoint changed, so we can prevent recurrence.
Project / service IDs: 878517e6-dffb-4f8c-a94d-516d30431593
Region/environment: REDIS US East
1 Replies
a month ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • about 1 month ago
a month ago
Can't tell you why your Redis restarted (only Railway staff can pull that from the service ID), but I can answer #2 and #3 and they're the interesting ones.
#2, yes. That's your whole outage. On Railway the private network hands out a new internal IPv6 address every time a container starts. The DNS name (redis.railway.internal) stays the same, but what it points at changed. Your Api and workers had 2d12h of uptime, meaning they were still holding sockets to an address that no longer exists. That's exactly why the symptom mutated from "connect ETIMEDOUT" to "Command timed out": the pool eventually opened new sockets, but half of them were stale/half-open TCP that will happily accept a write and never reply. The Redis server being healthy in metrics and your app being broken at the same time is the classic signature.
#3, no, nothing is blocking you. The fact that restarting the app services fixes it proves the path is fine. Once your clients re-resolve DNS they're back.
Things worth changing so a Redis restart doesn't take you down for hours again:
Make sure ioredis is actually resolving IPv6. This bites a lot of people on Railway. Node defaults to IPv4 lookups and the private network is v6, so add family: 0 (or ?family=0 on the URL):
new Redis(process.env.REDIS_URL, {
family: 0,
keepAlive: 30000,
connectTimeout: 10000,
commandTimeout: 5000,
maxRetriesPerRequest: 3,
})
keepAlive is the one that kills the zombie sockets. Without it a dead peer can hang around for the OS default (minutes to hours) and you get exactly the "command sent, no reply, ever" behaviour you're describing. commandTimeout means the hung ones fail fast instead of piling up into 2500 errors/min.
Add a self-heal. Railway healthchecks only run at deploy time, they won't recycle a running container, so nothing was going to rescue you automatically. Simplest thing that works: count consecutive Redis failures in your health check, and after N in a row call redis.disconnect() to force a fresh connect (which re-resolves DNS), or just process.exit(1) with the restart policy set to always and let the platform bring you back clean.
Also worth doing while you're in there: you're on noeviction with maxmemory unlimited. At 99MB you're nowhere near trouble, but if that dataset grows the container hits its RAM ceiling and gets OOM-killed by the platform, which is one plausible cause of the restart you saw. Set a maxmemory a bit under your plan's limit so Redis fails predictably instead of the container dying. And if those 590K keys matter, turn on appendonly, since RDB restore means you silently lost every write since the last snapshot.
For #1 you'll need staff to check the Redis service's deployment events around 17:15. Look for whether it was a host migration versus a crash, that changes what you can do about it. Either way, treat "Redis will restart without warning" as a given and build for it, because on a single-instance Redis it will happen again.