9 months ago
Hello,
I am trying to use the new MongoDB replica feature.
I follow what's recommended on Railway's github and set the correct environment variables according to https://github.com/railwayapp-templates/mongo-replica-set/tree/main/exampleApps/node .
I use the public names of the three mongodb instances for MONGO_HOSTS. (I even did an dns lookup and tried with the IPs directly.)
Nevertheless. I get the following error:MongoNetworkError: getaddrinfo ENOTFOUND mongo1.railway.internal
I don't understand where that private URL even comes from, it's nowhere in my code.
In general, I am wondering if anyone managed to make Railway's Mongo Replica template work (I have seen only less than 100 downloads).
Cheers,
11 Replies
9 months ago
Thanks for the report.
I've added the issue to triage, we'll take a look.
Again thanks,
David
Status changed to Awaiting User Response railway[bot] • 9 months ago
8 months ago
I am also having this exact issue. I am trying to connect from my local Node app to my public MongoDB replica.
If I use directConnection=true
then it connects fine, but it also means it is not connected as a replica set.
The connection string is (in development, public server):
mongodb://user:pass@autorack.proxy.rlwy.net:41983/?directConnection=false&replicaSet=rs0
TopologyDescription {
type: 'ReplicaSetNoPrimary',
servers: Map(3) {
'mongo1.railway.internal:27017' => [ServerDescription],
'mongo2.railway.internal:27017' => [ServerDescription],
'mongo3.railway.internal:27017' => [ServerDescription]
},
stale: false,
compatible: true,
heartbeatFrequencyMS: 10000,
localThresholdMS: 15,
setName: 'rs0',
maxElectionId: new ObjectId('7fffffff0000000000000001'),
maxSetVersion: 1,
commonWireVersion: 0,
logicalSessionTimeoutMinutes: null
},
code: undefined,
[Symbol(errorLabels)]: Set(0) {},
[cause]: MongoNetworkError: getaddrinfo ENOTFOUND mongo1.railway.internal
Status changed to Awaiting Railway Response railway[bot] • 9 months ago
8 months ago
Hey y'all,
We are waiting for the template author to get back online so we can proceed to answer these queries. Sit tight.
Thanks,
Angelo
Status changed to Awaiting User Response railway[bot] • 9 months ago
8 months ago
Thread has changed to "Awaiting User Response"
Sitting tight.
Status changed to Awaiting Railway Response railway[bot] • 9 months ago
8 months ago
Hey everyone!! I'm so excited you all are using the Mongo replica set, or trying to anyway. I'm sorry you all are running into this, but I have a lead on the issue.
A little explanation -
After the nodes are deployed from the template, the init service runs a command to join them to the set. This is the command it executes (ref):
rs.initiate({
_id: "$REPLICA_SET_NAME",
members: [
{ _id: 0, host: "$MONGO_PRIMARY_HOST:$MONGO_PORT" },
{ _id: 1, host: "$MONGO_REPLICA_HOST:$MONGO_PORT" },
{ _id: 2, host: "$MONGO_REPLICA2_HOST:$MONGO_PORT" }
]
})
The MONGO_---_HOST
and MONGO_PORT
variables are replaced with the values of the private network.
What I'm seeing is, even though all of the nodes are assigned a public domain and accessible via the internet directly, the internal communication between the nodes is still pinned to the private network, which is of course not accessible over the internet (e.g. applications running outside of the Railway project).
There are a couple of ways I am thinking we could approach this, one of which I am going to test right away, and the other would be a code change in the platform which obviously is a bigger undertaking.
Give me another day, and I will have an answer for you all! Thanks so much for your patience. I was out sick last week, otherwise, I'd have jumped in much sooner.
Status changed to Awaiting User Response railway[bot] • 8 months ago
8 months ago
Alrighty, I have a solution!!
As I mentioned above, the issue is related to how the nodes are configured to communicate over the private network which is scoped to the Railway project in which it is deployed. When you attempt to connect to the cluster from outside of the Railway project, it fails because the private network is not available outside of the project.
To solve the issue with connecting to the cluster from your local machine, you will need to setup a tunnel into the private network. As luck would have it, Brody just published a tutorial that covers how to implement a Tailscale subnet router in your Railway project, which provides this exact functionality.
I have been able to follow the tutorial and confirm that it works like a charm to enable connectivity into the private network from my local machine. The one hiccup I personally ran into, was in my local machine's Tailscale app, I had to enable Use Tailscale subnets
within Tailscale's settings:
Please let me know if you all have any problems with this solution. A possible alternative, is to configure the cluster to communicate over the public network, but that is inadvisable due to egress fees.
Attachments
8 months ago
Hi, thanks for the feedback. I almost got it running. I've setup Tailscale and everything needed to be able to use the MONGO_URI (composed of internal replica urls) and when I try to connect the error is thrown:
TopologyDescription {
type: 'ReplicaSetNoPrimary',
servers: Map(3) {
'mongo1.railway.internal:27017' => [ServerDescription],
'mongo2.railway.internal:27017' => [ServerDescription],
'mongo3.railway.internal:27017' => [ServerDescription]
},
stale: false,
compatible: true,
heartbeatFrequencyMS: 10000,
localThresholdMS: 15,
setName: 'rs0',
maxElectionId: null,
maxSetVersion: null,
commonWireVersion: 0,
logicalSessionTimeoutMinutes: null
},
code: undefined,
[Symbol(errorLabels)]: Set(0) {},
[cause]: MongoNetworkError: getaddrinfo EAI_AGAIN mongo1.railway.internal
at connectionFailureError (/home/bruno/Desktop/private/typescript/mern-monorepo-starter/node_modules/.pnpm/mongodb@6.8.0/node_modules/mongodb/lib/cmap/connect.js:353:20)
at Socket.<anonymous> (/home/bruno/Desktop/private/typescript/mern-monorepo-starter/node_modules/.pnpm/mongodb@6.8.0/node_modules/mongodb/lib/cmap/connect.js:268:44)
at Object.onceWrapper (node:events:634:26)
at Socket.emit (node:events:519:28)
at emitErrorNT (node:internal/streams/destroy:169:8)
at emitErrorCloseNT (node:internal/streams/destroy:128:3)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
[Symbol(errorLabels)]: Set(1) { 'ResetPool' },
[cause]: Error: getaddrinfo EAI_AGAIN mongo1.railway.internal
at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:118:26) {
errno: -3001,
code: 'EAI_AGAIN',
syscall: 'getaddrinfo',
hostname: 'mongo1.railway.internal'
}
}
}
[2024-08-31 12:18:06] error: getaddrinfo EAI_AGAIN mongo1.railway.internal
Status changed to Awaiting Railway Response railway[bot] • 8 months ago
8 months ago
hmm seems like the replication is in a bad state, do you mind if I connect to the services and troubleshoot?
Status changed to Awaiting User Response railway[bot] • 8 months ago
8 months ago
I was able to reproduce a similar error by forcing all of the nodes to restart at the same time. It seems that the replicaSet configuration is lost when this happens. I am going to look into why this happens, because it should be stored in the local
database which is persisted to the volume.
In the meantime, you should be able to connect to one of the nodes from a mongo shell and re-initialize the set by running:
rs.initiate({
_id: "rs0",
members: [
{ _id: 0, host: "mongo1.railway.internal:27017" },
{ _id: 1, host: "mongo2.railway.internal:27017" },
{ _id: 2, host: "mongo3.railway.internal:27017" }
]
})
The node on which you run this will become the primary. Let me know if that gets you unblocked. If not, I'd be happy to jump in and poke around if you give me a thumbs up.
3 months ago
Is there any way to customize template to run only with a single replica set? I know that is not recommended for production purposes, etc. This it's just for a test and development environment and to allow Prisma connections.
Status changed to Awaiting Railway Response railway[bot] • 3 months ago
3 months ago
Hello tiagoboeing,
Little bit off topic, but to answer your question, yes it should be possible, please open your own thread asking the question and we will help you over there!
Best,
Brody
Status changed to Awaiting User Response railway[bot] • 3 months ago