6 months ago
Project: Leads dashboard (481d9070-65d1-4d1f-a83a-798d29bc7e66)
Environment: production (388f2aa9-2631-4a3c-a5cd-35cf84d5e916)
Service: MongoDB (0b598554-1b44-4e47-96de-b5b112f3f8ec)
Issue: MongoDB 8.0 service is unable to start and continuously crashes with the following error:
MongoDB cannot start: Linux kernel versions 6.19 and newer has a known incompatibility with this version of MongoDB. See https://jira.mongodb.org/browse/SERVER-121912 for more information.
Details:
Image: mongo:8.0
Region: europe-west4-drams3a
Restart attempts: Multiple (does not resolve the issue)
Root cause: The region's Linux kernel version is 6.19 or newer, which has a known incompatibility with MongoDB 8.0 (MongoDB Jira: SERVER-121912)
Request: Please either:
Confirm if this region can be migrated to infrastructure with kernel < 6.19, or
Provide guidance on alternative regions that support MongoDB 8.0
4 Replies
Status changed to Open Railway • 6 months ago
0x5b62656e5d
Try changing the Mongo’s region to US West.
6 months ago
this is a temporary fix, is this issue also temporary?
samy-f
this is a temporary fix, is this issue also temporary?
6 months ago
I’m checking with the team. Did changing the region fix the issue?
a month ago
This isn't a Railway fault, and restarting won't fix it. It's MongoDB's own start-up guard. TCMalloc in MongoDB 8.x is broken on Linux kernels 6.19 to 7.0.13 (SERVER-121912), so mongod exits on purpose when it sees one of those kernels. Even mongod --version refuses. Moving region only helps while that region's hosts run an older kernel.
Fix that worked (tested today): add one variable to the MongoDB service and redeploy:
GLIBC_TUNABLES=glibc.pthread.rseq=1How I tested it: 9 GitHub-hosted runners on kernels 6.8, 6.17 and 7.0.0, with the official images. Each image had to start, then take a 20,000-document insert load.
| Image | Kernel 6.8 / 6.17 | Kernel 7.0.0, no variable | 7.0.0 + rseq=0 | 7.0.0 + rseq=1 |
|---|---|---|---|---|
| mongo:8.0.32, mongo:8.0 (8.0.34) | starts | refused | refused | starts, load OK |
| mongo:8.3 (8.3.13) | starts | refused | refused | starts, load OK |
| mongo:8.2 (8.2.12) | starts | starts | starts | starts |
| mongo:7.0 (7.0.45) | starts | starts | starts | starts |
(Apologies, I don't know how to format a table!)
Notes:
- Not
rseq=0. It's often suggested, but it was refused on every 8.0 and 8.3 run. - The variable is harmless on unaffected kernels. With it set, every image also started on 6.8 and 6.17, so you can leave it set and stop caring which host you land on.
- Other options I tested less:
mongo:8.2started everywhere without the variable. Check MongoDB's upgrade notes before moving existing 8.0 data onto it, because I didn't test an in-place upgrade. Don't point an 8.0 data volume atmongo:7.0without loweringfeatureCompatibilityVersionfirst. - To see your host's kernel: run
db.hostInfo().extra.kernelVersionin mongosh. Note thatos.versionis the distro release, not the kernel. - Caveat: I measured that the variable works. I haven't confirmed why, or whether it costs allocator performance under heavy load. The load test was short.
Test run (public logs): https://github.com/JKershaw/LinearViewer/actions/runs/39302841645