Container deployment errors after incident
jgak
HOBBYOP

a month ago

My application is not working properly after the incident. My backend authentication no longer works and I am getting the following errors :

2026-05-20T12:15:20.305755516Z [err]  ⚠️  SMTP variables detected as placeholders (e.g. "your-provider.com") → mock mode enabled.
2026-05-20T12:15:20.305761822Z [err]  ⚠️  Emails will not actually be sent.
2026-05-20T12:15:20.305768944Z [err]  ⚠️  Links (reset password, magic link, payment update) will be displayed here in the logs.
2026-05-20T12:15:20.310706696Z [inf]  [TTS] [OK] ElevenLabs TTS API key configured
2026-05-20T12:15:20.313843300Z [inf]  [STT] [OK] ElevenLabs STT API key configured
2026-05-20T12:15:20.313849501Z [inf]  [OK] Share privateRouter mounted on /api (POST/DELETE /chats/:id/share)
2026-05-20T12:15:20.313857519Z [inf]  [OK] Payment test pages: disabled (NODE_ENV=production). Set ENABLE_PAYMENT_TEST_PAGES=true to enable.
2026-05-20T12:15:20.313863717Z [inf]  [OK] Public share router mounted on /api/public
2026-05-20T12:15:20.313869462Z [inf]  [OK] Serving React build from: /build
2026-05-20T12:15:20.820641408Z [inf]  [RAG] Connected to index "sefaria-torahgpt"
2026-05-20T12:15:20.820648589Z [inf]  [RAG] Pinecone RAG service initialized successfully
2026-05-20T12:15:23.975339731Z [inf]  Starting Container
2026-05-20T12:15:53.651853877Z [err]      compatible: true,
2026-05-20T12:15:53.651863237Z [err]      heartbeatFrequencyMS: 10000,
2026-05-20T12:15:53.651872030Z [err]      localThresholdMS: 15,
2026-05-20T12:15:53.651889154Z [err]      at async /app/server.js:326:5 {
2026-05-20T12:15:53.651900140Z [err]    errorLabelSet: Set(0) {},
2026-05-20T12:15:53.651904111Z [err]      at _handleConnectionErrors (/app/node_modules/mongoose/lib/connection.js:1165:11)
2026-05-20T12:15:53.651919030Z [err]    reason: TopologyDescription {
2026-05-20T12:15:53.651919644Z [inf]  [!]  Falling back to HTTP instead...
2026-05-20T12:15:53.651928066Z [err]      at NativeConnection.openUri (/app/node_modules/mongoose/lib/connection.js:1096:11)
2026-05-20T12:15:53.651930520Z [err]      type: 'Unknown',
2026-05-20T12:15:53.651933986Z [err]  [ERROR] MongoDB connection failed: MongooseServerSelectionError: read ECONNRESET
2026-05-20T12:15:53.651939515Z [err]      servers: Map(1) { 'shuttle.proxy.rlwy.net:19852' => [ServerDescription] },
2026-05-20T12:15:53.651946397Z [err]      stale: false,
2026-05-20T12:15:53.651959542Z [err]    },
2026-05-20T12:15:53.651961032Z [err]  [ERROR] SSL certificates not found in server/ssl/
2026-05-20T12:15:53.651972446Z [err]     Run: ./generate-ssl-certs.sh (Linux/Mac) or .\generate-ssl-certs.ps1 (Windows)
2026-05-20T12:15:53.651975476Z [err]    code: undefined
2026-05-20T12:15:53.651982355Z [err]      setName: null,
2026-05-20T12:15:53.651984369Z [inf]  [!]  HTTP Server running on http://0.0.0.0:5000
2026-05-20T12:15:53.651987933Z [err]  }
2026-05-20T12:15:53.651994989Z [err]      maxElectionId: null,
2026-05-20T12:15:53.652006839Z [err]      commonWireVersion: 0,
2026-05-20T12:15:53.652007614Z [err]      maxSetVersion: null,
2026-05-20T12:15:53.652014750Z [err]      logicalSessionTimeoutMinutes: null
2026-05-20T12:22:25.703594743Z [inf]  [API Config] Using REACT_APP_API_URL: https://***.up.railway.app
2026-05-20T12:22:25.703603634Z [inf]  [API] Using REACT_APP_API_URL: https://***.up.railway.app
2026-05-20T12:22:25.703616816Z [inf]  [API] Final API_BASE: https://***.up.railway.app
2026-05-20T12:22:25.703631353Z [inf]  [OK] SSR bundle loaded from server/ssr/dist/react-ssr.cjs
2026-05-20T12:22:25.714238349Z [err]  [maintenance] cache refresh failed: Operation `appsettings.findOneAndUpdate()` buffering timed out after 10000ms
2026-05-20T12:22:25.722795926Z [err]  GET /api/ads/next failed: Operation `appsettings.findOneAndUpdate()` buffering timed out after 10000ms
$10 Bounty

7 Replies

Railway
BOT

a month ago

We experienced a major service disruption earlier today that affected traffic, TCP proxies, and other platform components. It was resolved at 07:57 UTC. Your MongoDB connection error (ECONNRESET via shuttle.proxy.rlwy.net) is consistent with residual effects from that incident. Your MongoDB service's last deployment was back in late April, so redeploying it (and then redeploying your TorahGPT backend) should re-establish the connection.


Status changed to Awaiting User Response Railway about 1 month ago


Railway

We experienced a [major service disruption](https://status.railway.com/incident/I23M92U0) earlier today that affected traffic, TCP proxies, and other platform components. It was resolved at 07:57 UTC. Your MongoDB connection error (`ECONNRESET` via `shuttle.proxy.rlwy.net`) is consistent with residual effects from that incident. Your MongoDB service's last deployment was back in late April, so redeploying it (and then redeploying your TorahGPT backend) should re-establish the connection.

jgak
HOBBYOP

a month ago

I redeployed MongoDB and TorahGPT, but I am still getting the same errors.


Status changed to Awaiting Railway Response Railway about 1 month ago


Status changed to Open Railway about 1 month ago


jgak

I redeployed MongoDB and TorahGPT, but I am still getting the same errors.

Have you tried redeploying all of your services?


0x5b62656e5d

Have you tried redeploying all of your services?

jgak
HOBBYOP

a month ago

Yes and I have errors with my Mongo deployment (deployed 15 min ago) :

2026-05-20T12:42:22.651052482Z [inf] Mounting volume on: /var/lib/containers/railwayapp/bind-mounts/938cc744-6a6c-4088-b618-b934d7d14a09/vol_vli528glotp6yg27

2026-05-20T12:42:22.651055947Z [inf] Starting Container

2026-05-20T12:42:23.354949368Z [inf] MongoDB cannot start: Linux kernel versions 6.19 and newer has a known incompatibility with this version of MongoDB. See https://jira.mongodb.org/browse/SERVER-121912 for more information.

2026-05-20T12:42:23.354954845Z [inf] MongoDB cannot start: Linux kernel versions 6.19 and newer has a known incompatibility with this version of MongoDB. See https://jira.mongodb.org/browse/SERVER-121912 for more information.

2026-05-20T12:42:23.653764933Z [inf] Mounting volume on: /var/lib/containers/railwayapp/bind-mounts/938cc744-6a6c-4088-b618-b934d7d14a09/vol_vli528glotp6yg27

2026-05-20T12:42:23.907367133Z [inf] MongoDB cannot start: Linux kernel versions 6.19 and newer has a known incompatibility with this version of MongoDB. See https://jira.mongodb.org/browse/SERVER-121912 for more information.

2026-05-20T12:42:24.525418204Z [inf] MongoDB cannot start: Linux kernel versions 6.19 and newer has a known incompatibility with this version of MongoDB. See https://jira.mongodb.org/browse/SERVER-121912 for more information.

2026-05-20T12:42:24.686950476Z [inf] Mounting volume on: /var/lib/containers/railwayapp/bind-mounts/938cc744-6a6c-4088-b618-b934d7d14a09/vol_vli528glotp6yg27

2026-05-20T12:42:24.686959650Z [inf] Mounting volume on: /var/lib/containers/railwayapp/bind-mounts/938cc744-6a6c-4088-b618-b934d7d14a09/vol_vli528glotp6yg27

2026-05-20T12:42:25.099330180Z [inf] MongoDB cannot start: Linux kernel versions 6.19 and newer has a known incompatibility with this version of MongoDB. See https://jira.mongodb.org/browse/SERVER-121912 for more information.

2026-05-20T12:42:25.664525985Z [inf] MongoDB cannot start: Linux kernel versions 6.19 and newer has a known incompatibility with this version of MongoDB. See https://jira.mongodb.org/browse/SERVER-121912 for more information.

2026-05-20T12:42:25.725809595Z [inf] Mounting volume on: /var/lib/containers/railwayapp/bind-mounts/938cc744-6a6c-4088-b618-b934d7d14a09/vol_vli528glotp6yg27

2026-05-20T12:42:25.725833992Z [inf] Mounting volume on: /var/lib/containers/railwayapp/bind-mounts/938cc744-6a6c-4088-b618-b934d7d14a09/vol_vli528glotp6yg27

2026-05-20T12:42:26.416150500Z [inf] MongoDB cannot start: Linux kernel versions 6.19 and newer has a known incompatibility with this version of MongoDB. See https://jira.mongodb.org/browse/SERVER-121912 for more information.

2026-05-20T12:42:26.749028285Z [inf] Mounting volume on: /var/lib/containers/railwayapp/bind-mounts/938cc744-6a6c-4088-b618-b934d7d14a09/vol_vli528glotp6yg27

2026-05-20T12:42:26.867411049Z [inf] MongoDB cannot start: Linux kernel versions 6.19 and newer has a known incompatibility with this version of MongoDB. See https://jira.mongodb.org/browse/SERVER-121912 for more information.

2026-05-20T12:42:27.461719595Z [inf] MongoDB cannot start: Linux kernel versions 6.19 and newer has a known incompatibility with this version of MongoDB. See https://jira.mongodb.org/browse/SERVER-121912 for more information.

2026-05-20T12:42:27.765102793Z [inf] Mounting volume on: /var/lib/containers/railwayapp/bind-mounts/938cc744-6a6c-4088-b618-b934d7d14a09/vol_vli528glotp6yg27

2026-05-20T12:42:27.765130394Z [inf] Mounting volume on: /var/lib/containers/railwayapp/bind-mounts/938cc744-6a6c-4088-b618-b934d7d14a09/vol_vli528glotp6yg27

2026-05-20T12:42:28.091012676Z [inf] MongoDB cannot start: Linux kernel versions 6.19 and newer has a known incompatibility with this version of MongoDB. See https://jira.mongodb.org/browse/SERVER-121912 for more information.

2026-05-20T12:42:28.661644373Z [inf] MongoDB cannot start: Linux kernel versions 6.19 and newer has a known incompatibility with this version of MongoDB. See https://jira.mongodb.org/browse/SERVER-121912 for more information.

2026-05-20T12:42:28.785188651Z [inf] Mounting volume on: /var/lib/containers/railwayapp/bind-mounts/938cc744-6a6c-4088-b618-b934d7d14a09/vol_vli528glotp6yg27

2026-05-20T12:42:28.785237314Z [inf] Mounting volume on: /var/lib/containers/railwayapp/bind-mounts/938cc744-6a6c-4088-b618-b934d7d14a09/vol_vli528glotp6yg27


jgak
HOBBYOP

a month ago

And after redeploying my app (second test), same errors :

2026-05-20T12:15:20.305755516Z [err] ⚠️ SMTP variables detected as placeholders (e.g. "your-provider.com") → mock mode enabled.

2026-05-20T12:15:20.305761822Z [err] ⚠️ Emails will not actually be sent.

2026-05-20T12:15:20.305768944Z [err] ⚠️ Links (reset password, magic link, payment update) will be displayed here in the logs.

2026-05-20T12:15:20.310706696Z [inf] [TTS] [OK] ElevenLabs TTS API key configured

2026-05-20T12:15:20.313843300Z [inf] [STT] [OK] ElevenLabs STT API key configured

2026-05-20T12:15:20.313849501Z [inf] [OK] Share privateRouter mounted on /api (POST/DELETE /chats/:id/share)

2026-05-20T12:15:20.313857519Z [inf] [OK] Payment test pages: disabled (NODE_ENV=production). Set ENABLE_PAYMENT_TEST_PAGES=true to enable.

2026-05-20T12:15:20.313863717Z [inf] [OK] Public share router mounted on /api/public

2026-05-20T12:15:20.313869462Z [inf] [OK] Serving React build from: /build

2026-05-20T12:15:20.820641408Z [inf] [RAG] Connected to index "sefaria-torahgpt"

2026-05-20T12:15:20.820648589Z [inf] [RAG] Pinecone RAG service initialized successfully

2026-05-20T12:15:23.975339731Z [inf] Starting Container

2026-05-20T12:15:53.651853877Z [err] compatible: true,

2026-05-20T12:15:53.651863237Z [err] heartbeatFrequencyMS: 10000,

2026-05-20T12:15:53.651872030Z [err] localThresholdMS: 15,

2026-05-20T12:15:53.651889154Z [err] at async /app/server.js:326:5 {

2026-05-20T12:15:53.651900140Z [err] errorLabelSet: Set(0) {},

2026-05-20T12:15:53.651904111Z [err] at _handleConnectionErrors (/app/node_modules/mongoose/lib/connection.js:1165:11)

2026-05-20T12:15:53.651919030Z [err] reason: TopologyDescription {

2026-05-20T12:15:53.651919644Z [inf] [!] Falling back to HTTP instead...

2026-05-20T12:15:53.651928066Z [err] at NativeConnection.openUri (/app/node_modules/mongoose/lib/connection.js:1096:11)

2026-05-20T12:15:53.651930520Z [err] type: 'Unknown',

2026-05-20T12:15:53.651933986Z [err] [ERROR] MongoDB connection failed: MongooseServerSelectionError: read ECONNRESET

2026-05-20T12:15:53.651939515Z [err] servers: Map(1) { 'shuttle.proxy.rlwy.net:19852' => [ServerDescription] },

2026-05-20T12:15:53.651946397Z [err] stale: false,

2026-05-20T12:15:53.651959542Z [err] },

2026-05-20T12:15:53.651961032Z [err] [ERROR] SSL certificates not found in server/ssl/

2026-05-20T12:15:53.651972446Z [err] Run: ./generate-ssl-certs.sh (Linux/Mac) or .\generate-ssl-certs.ps1 (Windows)

2026-05-20T12:15:53.651975476Z [err] code: undefined

2026-05-20T12:15:53.651982355Z [err] setName: null,

2026-05-20T12:15:53.651984369Z [inf] [!] HTTP Server running on http://0.0.0.0:5000

2026-05-20T12:15:53.651987933Z [err] }

2026-05-20T12:15:53.651994989Z [err] maxElectionId: null,

2026-05-20T12:15:53.652006839Z [err] commonWireVersion: 0,

2026-05-20T12:15:53.652007614Z [err] maxSetVersion: null,

2026-05-20T12:15:53.652014750Z [err] logicalSessionTimeoutMinutes: null

2026-05-20T12:22:25.703594743Z [inf] [API Config] Using REACT_APP_API_URL: https://***.railway.app

2026-05-20T12:22:25.703603634Z [inf] [API] Using REACT_APP_API_URL: https://***.railway.app

2026-05-20T12:22:25.703616816Z [inf] [API] Final API_BASE: https://***.up.railway.app

2026-05-20T12:22:25.703631353Z [inf] [OK] SSR bundle loaded from server/ssr/dist/react-ssr.cjs

2026-05-20T12:22:25.714238349Z [err] [maintenance] cache refresh failed: Operation appsettings.findOneAndUpdate() buffering timed out after 10000ms

2026-05-20T12:22:25.722795926Z [err] GET /api/ads/next failed: Operation appsettings.findOneAndUpdate() buffering timed out after 10000ms


aayankali
FREE

a month ago

verify the MONGODB_URL env var is pointing to the correct Railway internal URL, and add proper retry logic with explicit timeout settings in the Mongoose config (serverSelectionTimeoutMS, connectTimeoutMS).

2 Second issue is that SMTP is running in mock mode because the environment variables still contain placeholder values like your-provider.com. This means password reset emails, magic links, and payment update emails are not being delivered to users at all — they're only showing up in logs. The SMTP env vars (SMTP_HOST, SMTP_PORT, SMTP_USER, SMTP_PASS) need to be set to real credentials.

3 SSL certificates are missing from server/ssl/, so the server is falling back to plain HTTP on port 5000. The cert generation script (generate-ssl-certs.sh on Linux/Mac or generate-ssl-certs.ps1 on Windows) needs to be run and the output mounted at that path, or SSL should be handled at the Railway proxy level.

4 the React frontend has REACT_APP_API_URL pointing to torahgpt-backend-dev.up.railway.app even though NODE_ENV is set to production. All frontend API calls are hitting the dev backend, which needs to be corrected in the production environment variables.


jgak

And after redeploying my app (second test), same errors : 2026-05-20T12:15:20.305755516Z [err] ⚠️ SMTP variables detected as placeholders (e.g. "your-provider.com") → mock mode enabled. 2026-05-20T12:15:20.305761822Z [err] ⚠️ Emails will not actually be sent. 2026-05-20T12:15:20.305768944Z [err] ⚠️ Links (reset password, magic link, payment update) will be displayed here in the logs. 2026-05-20T12:15:20.310706696Z [inf] [TTS] [OK] ElevenLabs TTS API key configured 2026-05-20T12:15:20.313843300Z [inf] [STT] [OK] ElevenLabs STT API key configured 2026-05-20T12:15:20.313849501Z [inf] [OK] Share privateRouter mounted on /api (POST/DELETE /chats/:id/share) 2026-05-20T12:15:20.313857519Z [inf] [OK] Payment test pages: disabled (NODE_ENV=production). Set ENABLE_PAYMENT_TEST_PAGES=true to enable. 2026-05-20T12:15:20.313863717Z [inf] [OK] Public share router mounted on /api/public 2026-05-20T12:15:20.313869462Z [inf] [OK] Serving React build from: /build 2026-05-20T12:15:20.820641408Z [inf] [RAG] Connected to index "sefaria-torahgpt" 2026-05-20T12:15:20.820648589Z [inf] [RAG] Pinecone RAG service initialized successfully 2026-05-20T12:15:23.975339731Z [inf] Starting Container 2026-05-20T12:15:53.651853877Z [err] compatible: true, 2026-05-20T12:15:53.651863237Z [err] heartbeatFrequencyMS: 10000, 2026-05-20T12:15:53.651872030Z [err] localThresholdMS: 15, 2026-05-20T12:15:53.651889154Z [err] at async /app/server.js:326:5 { 2026-05-20T12:15:53.651900140Z [err] errorLabelSet: Set(0) {}, 2026-05-20T12:15:53.651904111Z [err] at _handleConnectionErrors (/app/node_modules/mongoose/lib/connection.js:1165:11) 2026-05-20T12:15:53.651919030Z [err] reason: TopologyDescription { 2026-05-20T12:15:53.651919644Z [inf] [!] Falling back to HTTP instead... 2026-05-20T12:15:53.651928066Z [err] at NativeConnection.openUri (/app/node_modules/mongoose/lib/connection.js:1096:11) 2026-05-20T12:15:53.651930520Z [err] type: 'Unknown', 2026-05-20T12:15:53.651933986Z [err] [ERROR] MongoDB connection failed: MongooseServerSelectionError: read ECONNRESET 2026-05-20T12:15:53.651939515Z [err] servers: Map(1) { 'shuttle.proxy.rlwy.net:19852' => [ServerDescription] }, 2026-05-20T12:15:53.651946397Z [err] stale: false, 2026-05-20T12:15:53.651959542Z [err] }, 2026-05-20T12:15:53.651961032Z [err] [ERROR] SSL certificates not found in server/ssl/ 2026-05-20T12:15:53.651972446Z [err] Run: ./generate-ssl-certs.sh (Linux/Mac) or .\generate-ssl-certs.ps1 (Windows) 2026-05-20T12:15:53.651975476Z [err] code: undefined 2026-05-20T12:15:53.651982355Z [err] setName: null, 2026-05-20T12:15:53.651984369Z [inf] [!] HTTP Server running on http://0.0.0.0:5000 2026-05-20T12:15:53.651987933Z [err] } 2026-05-20T12:15:53.651994989Z [err] maxElectionId: null, 2026-05-20T12:15:53.652006839Z [err] commonWireVersion: 0, 2026-05-20T12:15:53.652007614Z [err] maxSetVersion: null, 2026-05-20T12:15:53.652014750Z [err] logicalSessionTimeoutMinutes: null 2026-05-20T12:22:25.703594743Z [inf] [API Config] Using REACT_APP_API_URL: https://***.railway.app 2026-05-20T12:22:25.703603634Z [inf] [API] Using REACT_APP_API_URL: https://***.railway.app 2026-05-20T12:22:25.703616816Z [inf] [API] Final API_BASE: https://***.up.railway.app 2026-05-20T12:22:25.703631353Z [inf] [OK] SSR bundle loaded from server/ssr/dist/react-ssr.cjs 2026-05-20T12:22:25.714238349Z [err] [maintenance] cache refresh failed: Operation `appsettings.findOneAndUpdate()` buffering timed out after 10000ms 2026-05-20T12:22:25.722795926Z [err] GET /api/ads/next failed: Operation `appsettings.findOneAndUpdate()` buffering timed out after 10000ms

Try changing the Mongo instance to US West for now. See if that fixes it.


Welcome!

Sign in to your Railway account to join the conversation.

Loading...