Redis failing and timing out
arrow-multidevs
PROOP

a month ago

We are facing timeout on all our redis services:

10:05:36

redis 05:05:35.94 INFO  ==> Initializing Redis

10:05:36

redis 05:05:35.97 INFO  ==> Setting Redis config file

10:05:36

redis 05:05:36.00 INFO  ==> ** Redis setup finished! **


We had to wipe the attached volumes to get the services to work again. But once that was one, our system is unable to received any webhooks. We did not change any credentials or configs in our services.

Reference logs:

Webhook processing failed: 382 | throw new types_1.ValidationError('Missing custom auth', eventId);

383 | return undefined;

384 | }

385 | const expected = this.signCustom(unsigned);

386 | if (!this.safeCompare(expected, auth)) {

387 | throw new types_1.AuthenticationError('Invalid custom data signature', eventId);

                                ^

AuthenticationError: Invalid custom data signature

eventId: "order_created_8457801",

organizationId: undefined,

   code: "AUTHENTICATION_ERROR"

  at validateCustomData (/app/dist/billing/services/lemonsqueezy.service.js:387:31)

  at processSuccessfulPayment (/app/dist/billing/services/lemonsqueezy.service.js:247:34)

  at confirmWebhook (/app/dist/billing/services/lemonsqueezy.service.js:173:18)

  at handleWebhook (/app/dist/billing/controllers/lemonsqueezy.controller.js:22:38)

  at processTicksAndRejections (unknown:7:39)

Please help us with this situation.

$20 Bounty

2 Replies

Railway
BOT

a month ago

This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.

Status changed to Open Railway about 1 month ago


I'd start tracing your auth methods to see where went wrong.

If any secrets were stored in Redis, you'd need to set the values again.


a month ago

The Redis wipe and the webhook auth error may be connected only if your app stored webhook auth state/secrets in Redis.

The important line is ValidationError('Missing custom auth'), not the Redis startup log. I’d check where custom auth is loaded from. If it was cached/stored in Redis, wiping the volume erased it and you need to recreate those auth entries or re-run whatever setup seeds them.

Also check that the webhook sender is still sending the auth header after the recovery. If the header is missing on incoming requests, Redis is probably just a side effect, not the root cause.


Welcome!

Sign in to your Railway account to join the conversation.

Loading...