I am trying to mount an EMQX Cluster using HAProxy as a reverse proxy, however the HAProxy service keeps failing to start. This is the content of my haproxy.cfg \[Due to the character limit, I placed on my gist\]: This is the content of my logs \[Find the full dump here \]: `[WARNING] (1) : Server mqtt_backend/emqx1 is DOWN, reason: Layer4 connection problem, info: "Connection refused at initial connection step of tcp-check", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.` `[ALERT] (1) : backend 'mqtt_backend' has no server available!` What I'm I doing wrong?

EMQX Cluster + HAProxy keeps failing to start, what do I do?

Status changed to Open Railway • 20 days ago

i-smuglov

FREETop 5% Contributor

19 days ago

Let's start by verifying env vars actually resolve:

echo $EMQX1_HOST

i-smuglov

Let's start by verifying env vars actually resolve: ``` echo $EMQX1_HOST ```

amihub

PROOP

19 days ago

root@5670edcfaad9:/var/lib/haproxy# echo $EMQX1_HOST

emqx1.railway.internal

amihub

root@5670edcfaad9:/var/lib/haproxy# echo $EMQX1\_HOST emqx1.railway.internal

i-smuglov

FREETop 5% Contributor

19 days ago

What’s wrong

On Railway, services do not expose ports to each other unless they are actually listening internally.

So either:

${EMQX_MQTT_PORT} is wrong
or EMQX is not bound to 0.0.0.0

Do this (fastest fix)

1. Stop using env port, hardcode:

server emqx1 emqx1.railway.internal:1883 check

2. Ensure EMQX listens on all interfaces:

listener.tcp.default = 0.0.0.0:1883

Verify from HAProxy container

nc -zv emqx1.railway.internal 1883

FAIL → EMQX not listening / wrong port
SUCCESS → HAProxy will work immediately

Most likely cause

Your ${EMQX_MQTT_PORT} ≠ actual EMQX port (very common on Railway).

amihub

PROOP

19 days ago

I am a lot more confused than I was now.

i-smuglov

### What’s wrong On Railway, services **do not expose ports to each other unless they are actually listening internally**. So either: * `${EMQX_MQTT_PORT}` is wrong * or EMQX is not bound to `0.0.0.0` ### Do this (fastest fix) **1\. Stop using env port, hardcode:** ``` server emqx1 emqx1.railway.internal:1883 check ``` **2\. Ensure EMQX listens on all interfaces:** ``` listener.tcp.default = 0.0.0.0:1883 ``` ### Verify from HAProxy container ``` nc -zv emqx1.railway.internal 1883 ``` * FAIL → EMQX not listening / wrong port * SUCCESS → HAProxy will work immediately ### Most likely cause Your `${EMQX_MQTT_PORT}` ≠ actual EMQX port (very common on Railway).

amihub

PROOP

19 days ago

I am a lot more confused than I was now.

amihub

I am a lot more confused than I was now.

i-smuglov

FREETop 5% Contributor

19 days ago

The situation is simpler than it looks. Ignore HAProxy for a moment.

HAProxy can find your EMQX container, but when it tries to connect:

the port is closed
or EMQX is not listening on that port

Run this inside your HAProxy container:

nc -zv emqx1.railway.internal 1883

Result interpretation

Connection refused → EMQX is not listening on 1883
Connected → HAProxy config is fine

i-smuglov

The situation is simpler than it looks. Ignore HAProxy for a moment. ### HAProxy **can find your EMQX container**, but when it tries to connect: * the port is **closed** * or EMQX is **not listening on that port** Run this inside your HAProxy container: ``` nc -zv emqx1.railway.internal 1883 ``` Result interpretation * **Connection refused** → EMQX is not listening on 1883 * **Connected** → HAProxy config is fine

amihub

PROOP

19 days ago

root@db3647e88a60:/var/lib/haproxy# nc -zv emqx1.railway.internal 1883

bash: nc: command not found

i-smuglov

The situation is simpler than it looks. Ignore HAProxy for a moment. ### HAProxy **can find your EMQX container**, but when it tries to connect: * the port is **closed** * or EMQX is **not listening on that port** Run this inside your HAProxy container: ``` nc -zv emqx1.railway.internal 1883 ``` Result interpretation * **Connection refused** → EMQX is not listening on 1883 * **Connected** → HAProxy config is fine

amihub

PROOP

19 days ago

nc: connect to emqx1.railway.internal (fd12:dafa:1c10:0:4000:12f:18d1:9312) port 1883 (tcp) failed: Connection refused

amihub

nc: connect to emqx1.railway.internal (fd12:dafa:1c10:0:4000:12f:18d1:9312) port 1883 (tcp) failed: Connection refused

i-smuglov

FREETop 5% Contributor

19 days ago

1. Ensure MQTT listener is enabled and external

In your EMQX config:

listener.tcp.default = 0.0.0.0:1883

2. Check if EMQX is actually running

Inside the EMQX container:

emqx ctl status

3. Verify the port inside EMQX container

ss -lntp | grep 1883

You MUST see:

0.0.0.0:1883

If:

nothing shows → MQTT listener is disabled
127.0.0.1:1883 → not accessible from HAProxy

i-smuglov

### 1\. Ensure MQTT listener is enabled and external In your EMQX config: ``` listener.tcp.default = 0.0.0.0:1883 ``` ### 2\. Check if EMQX is actually running Inside the EMQX container: ``` emqx ctl status ``` ### 3\. Verify the port inside EMQX container ``` ss -lntp | grep 1883 ``` You MUST see: ``` 0.0.0.0:1883 ``` If: * nothing shows → MQTT listener is disabled * `127.0.0.1:1883` → not accessible from HAProxy

amihub

PROOP

19 days ago

1. Ensure MQTT listener is enabled and external

In your EMQX config:

listener.tcp.default = 0.0.0.0:1883
result ==> ## NOTE:
## This config file overrides data/configs/cluster.hocon,
## and is merged with environment variables which start with 'EMQX_' prefix.
##
## Config changes made from EMQX dashboard UI, management HTTP API, or CLI
## are stored in data/configs/cluster.hocon.
## To avoid confusion, please do not store the same configs in both files.
##
## See https://www.emqx.io/docs/en/v5.0/configuration/configuration.html for more details.
## Configuration full example can be found in etc/examples

node {
  name = "emqx@127.0.0.1"
  cookie = "xxxxxxxxxxxxxxxx"
  data_dir = "data"
}

cluster {
  name = emqxcl
  discovery_strategy = manual
}

dashboard {
    listeners.http {
        bind = 18083
    }
}

2. Check if EMQX is actually running

Inside the EMQX container:

emqx ctl status
Result ==> Node 'emqx@emqx1.railway.internal' not responding to pings.

3. Verify the port inside EMQX container

ss -lntp | grep 1883
Result ==> LISTEN 0      1024         0.0.0.0:1883       0.0.0.0:*

amihub

### **1\. Ensure MQTT listener is enabled and external** In your EMQX config: ``` listener.tcp.default = 0.0.0.0:1883 result ==> ## NOTE: ## This config file overrides data/configs/cluster.hocon, ## and is merged with environment variables which start with 'EMQX_' prefix. ## ## Config changes made from EMQX dashboard UI, management HTTP API, or CLI ## are stored in data/configs/cluster.hocon. ## To avoid confusion, please do not store the same configs in both files. ## ## See https://www.emqx.io/docs/en/v5.0/configuration/configuration.html for more details. ## Configuration full example can be found in etc/examples node { name = "emqx@127.0.0.1" cookie = "xxxxxxxxxxxxxxxx" data_dir = "data" } cluster { name = emqxcl discovery_strategy = manual } dashboard { listeners.http { bind = 18083 } } ``` ### **2\. Check if EMQX is actually running** Inside the EMQX container: ``` emqx ctl status Result ==> Node 'emqx@emqx1.railway.internal' not responding to pings. ``` **3\. Verify the port inside EMQX container** ``` ss -lntp | grep 1883 Result ==> LISTEN 0 1024 0.0.0.0:1883 0.0.0.0:* ```

i-smuglov

FREETop 5% Contributor

19 days ago

Try to set node name to match Railway hostname:

node {
  name = "emqx@emqx1.railway.internal"
}

Or better (portable):

node {
  name = "emqx@${HOSTNAME}"
}

Then restart EMQX

After restart, verify:

emqx ctl status

You should see:

Node 'emqx@emqx1.railway.internal' is running

amihub

### **1\. Ensure MQTT listener is enabled and external** In your EMQX config: ``` listener.tcp.default = 0.0.0.0:1883 result ==> ## NOTE: ## This config file overrides data/configs/cluster.hocon, ## and is merged with environment variables which start with 'EMQX_' prefix. ## ## Config changes made from EMQX dashboard UI, management HTTP API, or CLI ## are stored in data/configs/cluster.hocon. ## To avoid confusion, please do not store the same configs in both files. ## ## See https://www.emqx.io/docs/en/v5.0/configuration/configuration.html for more details. ## Configuration full example can be found in etc/examples node { name = "emqx@127.0.0.1" cookie = "xxxxxxxxxxxxxxxx" data_dir = "data" } cluster { name = emqxcl discovery_strategy = manual } dashboard { listeners.http { bind = 18083 } } ``` ### **2\. Check if EMQX is actually running** Inside the EMQX container: ``` emqx ctl status Result ==> Node 'emqx@emqx1.railway.internal' not responding to pings. ``` **3\. Verify the port inside EMQX container** ``` ss -lntp | grep 1883 Result ==> LISTEN 0 1024 0.0.0.0:1883 0.0.0.0:* ```

i-smuglov

FREETop 5% Contributor

17 days ago

Any luck?

Welcome!

Sign in to your Railway account to join the conversation.

Login