Skip to content
MongoDB 8.0 fails to start on kernel 6.19+ (europe-west4-drams3a region)
samy-f
OP

6 months ago

Project: Leads dashboard (481d9070-65d1-4d1f-a83a-798d29bc7e66)

Environment: production (388f2aa9-2631-4a3c-a5cd-35cf84d5e916)

Service: MongoDB (0b598554-1b44-4e47-96de-b5b112f3f8ec)

Issue: MongoDB 8.0 service is unable to start and continuously crashes with the following error:

MongoDB cannot start: Linux kernel versions 6.19 and newer has a known incompatibility with this version of MongoDB. See https://jira.mongodb.org/browse/SERVER-121912 for more information.

Details:

Image: mongo:8.0

Region: europe-west4-drams3a

Restart attempts: Multiple (does not resolve the issue)

Root cause: The region's Linux kernel version is 6.19 or newer, which has a known incompatibility with MongoDB 8.0 (MongoDB Jira: SERVER-121912)

Request: Please either:

Confirm if this region can be migrated to infrastructure with kernel < 6.19, or

Provide guidance on alternative regions that support MongoDB 8.0

$20 Bounty

4 Replies

Status changed to Open Railway • 6 months ago


Try changing the Mongo’s region to US West.


0x5b62656e5d

Try changing the Mongo’s region to US West.

samy-f
OP

6 months ago

this is a temporary fix, is this issue also temporary?


samy-f

this is a temporary fix, is this issue also temporary?

I’m checking with the team. Did changing the region fix the issue?


jkershaw

a month ago

This isn't a Railway fault, and restarting won't fix it. It's MongoDB's own start-up guard. TCMalloc in MongoDB 8.x is broken on Linux kernels 6.19 to 7.0.13 (SERVER-121912), so mongod exits on purpose when it sees one of those kernels. Even mongod --version refuses. Moving region only helps while that region's hosts run an older kernel.

Fix that worked (tested today): add one variable to the MongoDB service and redeploy:

GLIBC_TUNABLES=glibc.pthread.rseq=1

How I tested it: 9 GitHub-hosted runners on kernels 6.8, 6.17 and 7.0.0, with the official images. Each image had to start, then take a 20,000-document insert load.

| Image | Kernel 6.8 / 6.17 | Kernel 7.0.0, no variable | 7.0.0 + rseq=0 | 7.0.0 + rseq=1 |

|---|---|---|---|---|

| mongo:8.0.32, mongo:8.0 (8.0.34) | starts | refused | refused | starts, load OK |

| mongo:8.3 (8.3.13) | starts | refused | refused | starts, load OK |

| mongo:8.2 (8.2.12) | starts | starts | starts | starts |

| mongo:7.0 (7.0.45) | starts | starts | starts | starts |

(Apologies, I don't know how to format a table!)

Notes:

  • Not rseq=0. It's often suggested, but it was refused on every 8.0 and 8.3 run.
  • The variable is harmless on unaffected kernels. With it set, every image also started on 6.8 and 6.17, so you can leave it set and stop caring which host you land on.
  • Other options I tested less: mongo:8.2 started everywhere without the variable. Check MongoDB's upgrade notes before moving existing 8.0 data onto it, because I didn't test an in-place upgrade. Don't point an 8.0 data volume at mongo:7.0 without lowering featureCompatibilityVersion first.
  • To see your host's kernel: run db.hostInfo().extra.kernelVersion in mongosh. Note that os.version is the distro release, not the kernel.
  • Caveat: I measured that the variable works. I haven't confirmed why, or whether it costs allocator performance under heavy load. The load test was short.

Test run (public logs): https://github.com/JKershaw/LinearViewer/actions/runs/39302841645


Welcome!

Sign in to your Railway account to join the conversation.

Loading...