a year ago
It seems our redis instance has failed. First we started experiencing heavy delays, and our services started failing to connect to redis. So I restarted redis and it still failed. Here's the logs leading up to the failure
1:C 09 Apr 2024 22:04:25.888 * Redis version=7.2.3, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 09 Apr 2024 22:04:25.888 * Configuration loaded
1:M 09 Apr 2024 22:04:25.888 * monotonic clock: POSIX clock_gettime
1:M 09 Apr 2024 22:04:25.893 * Running mode=standalone, port=6379.
1:M 09 Apr 2024 22:04:25.894 * Server initialized
1:M 09 Apr 2024 22:04:25.894 * Reading RDB base file on AOF loading…
1:M 09 Apr 2024 22:04:25.894 * Loading RDB produced by version 7.2.3
1:M 09 Apr 2024 22:04:25.894 * RDB age 18480 seconds
1:M 09 Apr 2024 22:04:25.894 * RDB memory usage when created 3.14 Mb
1:M 09 Apr 2024 22:04:25.894 * RDB is base AOF
1:M 09 Apr 2024 22:04:25.896 * Done loading RDB, keys loaded: 359, keys expired: 0.
1:M 09 Apr 2024 22:04:25.896 * DB loaded from base file appendonly.aof.129.base.rdb: 0.002 seconds
1:M 09 Apr 2024 22:04:25.975 # Bad file format reading the append only file appendonly.aof.129.incr.aof: make a backup of your AOF file, then use ./redis-check-aof --fix
redis 22:04:39.86 INFO ==>
redis 22:04:39.86 INFO ==> Welcome to the Bitnami redis container
redis 22:04:39.86 INFO ==> Subscribe to project updates by watching https://github.com/bitnami/containers
redis 22:04:39.86 INFO ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
redis 22:04:39.87 INFO ==>
1:C 09 Apr 2024 22:06:00.261 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 09 Apr 2024 22:06:00.262 * Redis version=7.2.3, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 09 Apr 2024 22:06:00.262 * Configuration loaded
1:M 09 Apr 2024 22:06:00.262 * monotonic clock: POSIX clock_gettime
1:M 09 Apr 2024 22:06:00.262 * Running mode=standalone, port=6379.
1:M 09 Apr 2024 22:06:00.263 * Server initialized
1:M 09 Apr 2024 22:06:00.263 * Reading RDB base file on AOF loading…
1:M 09 Apr 2024 22:06:00.263 * Loading RDB produced by version 7.2.3
1:M 09 Apr 2024 22:06:00.263 * RDB age 18575 seconds
1:M 09 Apr 2024 22:06:00.263 * RDB memory usage when created 3.14 Mb
1:M 09 Apr 2024 22:06:00.263 * RDB is base AOF
1:M 09 Apr 2024 22:06:00.265 * Done loading RDB, keys loaded: 359, keys expired: 0.
1:M 09 Apr 2024 22:06:00.265 * DB loaded from base file appendonly.aof.129.base.rdb: 0.002 seconds
1:M 09 Apr 2024 22:06:00.341 # Bad file format reading the append only file appendonly.aof.129.incr.aof: make a backup of your AOF file, then use ./redis-check-aof --fix
> ⓘ Deployment information is only viewable by Project members and Railway admins.
7 Replies
a year ago
Can you provide screenshots of the metrics for redis set to 6 hours? assuming the initial crash was less then 6 hours ago?
a year ago
@brody
Attachments
a year ago
I believe not, but not 100% sure. This redis service is strictly for n8n.
a year ago
Then it would just store queue data for the worker, something that would be auto generated.
Looks like the AOF file got corrupted, Railway doesn't take user accessable backups so you are left with wiping the volume on Redis, once wiped, redeploy the crashed deployment and then redeploy the worker service.