10 days ago
Starting yesterday, my service began crashing and restarting constantly. The logs show the same unhandled error repeated many times:
Error: read ECONNRESET
at TCP.onStreamRead (node:internal/stream_base_commons:218:20)
at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {errno: -104,
code: 'ECONNRESET',
syscall: 'read'
}
Error: read ECONNRESET
at TCP.onStreamRead (node:internal/stream_base_commons:218:20)
at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {errno: -104,
code: 'ECONNRESET',
syscall: 'read'
}
Error: read ECONNRESET
at TCP.onStreamRead (node:internal/stream_base_commons:218:20)
at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {errno: -104,
code: 'ECONNRESET',
syscall: 'read'
}
Error: read ECONNRESET
at TCP.onStreamRead (node:internal/stream_base_commons:218:20)
at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {errno: -104,
code: 'ECONNRESET',
syscall: 'read'
}
Error: read ECONNRESET
at TCP.onStreamRead (node:internal/stream_base_commons:218:20)
at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {errno: -104,
code: 'ECONNRESET',
syscall: 'read'
}
Error: read ECONNRESET
at TCP.onStreamRead (node:internal/stream_base_commons:218:20)
at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {errno: -104,
code: 'ECONNRESET',
syscall: 'read'
}
Error: read ECONNRESET
at TCP.onStreamRead (node:internal/stream_base_commons:218:20)
at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {errno: -104,
code: 'ECONNRESET',
syscall: 'read'
}
Error: read ECONNRESET
at TCP.onStreamRead (node:internal/stream_base_commons:218:20)
at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {errno: -104,
code: 'ECONNRESET',
syscall: 'read'
}
Error: read ECONNRESET
at TCP.onStreamRead (node:internal/stream_base_commons:218:20)
at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {errno: -104,
code: 'ECONNRESET',
syscall: 'read'
}
Error: read ECONNRESET
at TCP.onStreamRead (node:internal/stream_base_commons:218:20)
at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {errno: -104,
code: 'ECONNRESET',
syscall: 'read'
}
Error: read ECONNRESET
at TCP.onStreamRead (node:internal/stream_base_commons:218:20)
at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {errno: -104,
code: 'ECONNRESET',
syscall: 'read'
}
Error: read ECONNRESET
at TCP.onStreamRead (node:internal/stream_base_commons:218:20)
at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {errno: -104,
code: 'ECONNRESET',
syscall: 'read'
}
Some important context:
- I have NOT deployed or changed any code recently. The same build that ran fine before is now crashing.
- This never happened before — it started within the last day.
- The restarts were so frequent that I had to raise the restart limit to 200 just to keep the service alive.
1 Replies
10 days ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • 10 days ago
2 hours ago
That stack is the tell. There are zero frames from your code in it — it goes straight from TCP.onStreamRead to callbackTrampoline. Nothing awaited this, and no try/catch or .catch() could ever have caught it. It's an 'error' event emitted on a socket that has no listener attached, and Node's rule is that an EventEmitter emitting 'error' with no handler throws — which takes the process down.
So this isn't really a crash caused by ECONNRESET. It's a crash caused by one socket in your app not having an error handler. The ECONNRESET is just what finally triggered it.
That's also why "same build, no code changes" fits rather than contradicts. Something on the other end started closing connections — a managed database rolling a node, an upstream provider changing an idle timeout, a load balancer swap. The missing handler was probably there for months; the peer's behaviour is what changed in the last day.
To find which socket:
Set NODE_OPTIONS=--trace-uncaught as a service variable. That prints the full async stack for uncaught exceptions instead of the truncated one, and it usually names the library directly.
Usual culprits, roughly in order of likelihood:
- node-postgres: needs pool.on('error', err => ...). Idle pooled clients get killed server-side all the time and the Pool emits 'error' on itself. This is by far the most common cause of exactly this stack.
-
- ioredis / redis: client.on('error', ...)
-
- Raw net/tls sockets, or an http/https Agent with keepAlive: true reusing a socket the peer already closed
-
- ws or any WebSocket client: ws.on('error', ...)
Add the handler wherever it turns out to be. Also add a process.on('uncaughtException') that logs and then exits deliberately — so the next occurrence gives you a real log line with context instead of a bare stack.
One note on the restart limit at 200: that keeps you alive but every crash is still a cold start, so you're paying for it in latency. Once the handler is in, drop it back.
If keep-alive turns out to be the cause, set your agent's idle timeout below the peer's — otherwise you'll keep picking up sockets the far end has already reset.