22 days ago
We are facing timeout on all our redis services:
10:05:36
[38;5;6mredis [38;5;5m05:05:35.94 [0m[38;5;2mINFO [0m ==> Initializing Redis
10:05:36
[38;5;6mredis [38;5;5m05:05:35.97 [0m[38;5;2mINFO [0m ==> Setting Redis config file
10:05:36
[38;5;6mredis [38;5;5m05:05:36.00 [0m[38;5;2mINFO [0m ==> ** Redis setup finished! **
We had to wipe the attached volumes to get the services to work again. But once that was one, our system is unable to received any webhooks. We did not change any credentials or configs in our services.
Reference logs:
Webhook processing failed: 382 | throw new types_1.ValidationError('Missing custom auth', eventId);
383 | return undefined;
384 | }
385 | const expected = this.signCustom(unsigned);
386 | if (!this.safeCompare(expected, auth)) {
387 | throw new types_1.AuthenticationError('Invalid custom data signature', eventId);
^AuthenticationError: Invalid custom data signature
eventId: "order_created_8457801",organizationId: undefined,
code: "AUTHENTICATION_ERROR"
at validateCustomData (/app/dist/billing/services/lemonsqueezy.service.js:387:31)
at processSuccessfulPayment (/app/dist/billing/services/lemonsqueezy.service.js:247:34)
at confirmWebhook (/app/dist/billing/services/lemonsqueezy.service.js:173:18)
at handleWebhook (/app/dist/billing/controllers/lemonsqueezy.controller.js:22:38)
at processTicksAndRejections (unknown:7:39)Please help us with this situation.
2 Replies
22 days ago
This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.
Status changed to Open Railway • 22 days ago
22 days ago
I'd start tracing your auth methods to see where went wrong.
If any secrets were stored in Redis, you'd need to set the values again.
22 days ago
The Redis wipe and the webhook auth error may be connected only if your app stored webhook auth state/secrets in Redis.
The important line is ValidationError('Missing custom auth'), not the Redis startup log. I’d check where custom auth is loaded from. If it was cached/stored in Redis, wiping the volume erased it and you need to recreate those auth entries or re-run whatever setup seeds them.
Also check that the webhook sender is still sending the auth header after the recovery. If the header is missing on incoming requests, Redis is probably just a side effect, not the root cause.