a day ago
We forward logs from one environment (19 services) with Locomotive, which uses the environmentLogs GraphQL subscription. Sentry consistently received fewer lines than Railway has. We instrumented the subscriber to account for every line it receives, ran it for 70 minutes, and compared the result with railway logs --since … --until … --json for the same services and window.
Subscription used
subscription streamEnvironmentLogs($environmentId: String!, $filter: String, $beforeLimit: Int!, $beforeDate: String) {
environmentLogs(environmentId: $environmentId, filter: $filter, beforeDate: $beforeDate, beforeLimit: $beforeLimit) { … }
}filter = the service-id filter for the 19 services, beforeLimit: 5000, and beforeDate = the last timestamp seen. That's the resume point after a reconnect.
Result (2026-11-11, 16:39–17:49 UTC)
- The logs query returned 5,687 lines. The subscription delivered 5,617. 70 lines (1.2%) were never delivered.
- Every line the subscription delivered was accounted for: summed batch sizes = lines forwarded + duplicates. The subscriber dropped nothing.
- Reconnects weren't the cause. The server closed the stream with
4500 deadline_exceededabout every 10 minutes, plus a few EOFs, and each resubscribe resumed exactly atbeforeDate. - The missing lines arrive in the middle of a healthy stream. Batches covering their timestamps arrived, but those lines weren't in them. They're partial gaps inside bursts. For example, 12 of 38 lines from one service within one second are missing, and all 10 lines from another service at one moment are missing.
- We saw the same rate over a longer earlier window: ~1.6% over 15 hours on one service.
Expected: the subscription delivers the same lines the logs query returns for the same filter and window.
Questions:
- Is the subscription expected to be lossy, e.g. sampled or best-effort under bursts?
- If so, is there a reliable way to tail logs, such as a per-deployment subscription or a paginated query with a cursor?
We can provide the environment ID, the exact missing timestamps and the probe output privately.
2 Replies
Status changed to Awaiting Railway Response Railway • 1 day ago
21 hours ago
Thanks for the detailed investigation. The subscription isn’t intentionally sampled. We found a bug where logs that reached storage late could be skipped after the stream had advanced past their timestamps. Those lines would still appear in historical queries.
We’ve fixed the server-side handling and released Locomotive 0.6.2, which also fixes how Locomotive handles late logs and reconnects. These changes passed our independent tests reproducing those gaps.
Please update to 0.6.2 and repeat the comparison. Recovery is still bounded by time and batch limits, so this doesn’t provide an unlimited, lossless delivery guarantee. If you still see missing lines, please share the missing timestamps and probe output privately so we can investigate further.
Status changed to Awaiting User Response Railway • about 21 hours ago
21 minutes ago
Thanks for the quick fix. I can confirm the gap is gone.
We deployed Locomotive 0.6.2 on Nov 12 at 07:25 UTC. For three of our busiest services, I compared the line count from railway logs with what arrived through the subscription, hour by hour, until 15:00 UTC.
Before the fix, a 2-hour sample on Nov 11 was missing 292 of 11,951 lines (2.4%). After 0.6.2, none of the 34,871 lines from 07:30–15:00 UTC went missing from the stream. That includes bursts and the regular deadline_exceeded reconnects every ~10 minutes. At the old rate, we would have expected roughly 400–850 missing lines.
Status changed to Awaiting Railway Response Railway • 21 minutes ago
Status changed to Solved manuelgu • 21 minutes ago