15 days ago
On the managed Postgres service, log lines are classified by which stream they came from rather than by the level the line actually declares. Everything Postgres and pgBackRest emit lands on stderr, so routine output arrives tagged error. Same service, same minute, straight from the logs API:
severity: "error" → P00 INFO: archive-push command end: completed successfully (605ms)
severity: "error" → [128] LOG: checkpoint complete: wrote 69 buffers (0.4%) ...
severity: "info" → pgbackrest-watcher: iteration: no action (failed=0, lag=0)
The watcher line is the tell — it's the one thing here that writes to stdout, and it's the only one classified correctly.
Why this matters more than cosmetics. Postgres defaults to log_destination = stderr, so this isn't limited to backup chatter — a genuine ERROR:, FATAL: or PANIC: carries exactly the same severity as LOG: checkpoint starting. Severity therefore conveys nothing on this service, and the dashboard's error view can't be used to spot real problems.
Volume makes it concrete. With PITR enabled, archive_timeout is 60s, so a WAL segment ships every minute and each push emits three INFO lines — roughly 4,000 lines a day, all tagged error, on an idle database. Real errors are buried in that. I only found three genuine Postgres errors in my own logs by filtering the noise out by hand, and one of them turned out to be an actual application bug that had been invisible for hours.
Suggested fix: classify by the level the line declares, not by the stream it arrived on. Both formats are unambiguous and stable:
Postgres: UTC [pid] LEVEL: where LEVEL is DEBUG/INFO/NOTICE/WARNING/LOG/ERROR/FATAL/PANIC
pgBackRest: P00 INFO: / WARN: / ERROR:
Falling back to the stream-based guess only when no prefix matches would keep current behaviour for everything else.
Current workaround, in case it helps anyone else — this filter drops the routine noise and leaves real errors visible:
-"archive-push" -"pushed WAL file" -"checkpoint" -"pgbackrest-watcher"
It works, but it has to be retyped every time, and it silently hides a real ERROR: that happens to mention a checkpoint.
Attachments
0 Replies