COPY FROM STDIN fails on Railway Postgres with invalid byte sequence for encoding "UTF8": 0x00
alex-delia
PROOP

11 days ago

We insert Valorant match analytics using COPY ... FROM STDIN WITH (FORMAT csv) inside a transaction (BEGIN → multiple COPY into temp staging tables → INSERT ... SELECT merge → COMMIT).

The same application code, client libraries, and payloads succeed against our local PostgreSQL 17.10 instance but fail intermittently against our Railway PostgreSQL database with:

ERROR: invalid byte sequence for encoding "UTF8": 0x00

SQLSTATE: 22021

CONTEXT: COPY tmp_valorant_player_round, line 79

We have verified that the CSV row Postgres reports as failing does not contain a 0x00 byte in the data we generate or send. Application-side NUL preflight and stream guards do not fire. The failure appears specific to the remote Railway connection path, not to source data, CSV serialization, or a Postgres version difference.

This code worked previously with railway but it seems something potentially changed in the last couple days that are preventing this from working consistently.

$20 Bounty

5 Replies

Railway
BOT

11 days ago

This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.

Status changed to Open Railway 11 days ago


chandrakiran-gr
HOBBYTop 10% Contributor

10 days ago

I tried to reproduce this on a fresh Railway Postgres (Hobby) and couldn't trigger the 0x00 / 22021 error, which I think is itself a useful data point. Sharing the setup in case it helps you narrow it.

What I ran: a Python/psycopg3 harness doing the same BEGIN -> COPY FROM STDIN (csv) into a temp staging table -> commit loop, over the public proxy host (*.proxy.rlwy.net). The generated data is asserted NUL-free before every send, so any 0x00 the server reported would have to have appeared after my buffer. ~2.6M rows total across four configs: TLS persistent connection, plaintext (sslmode=disable), fresh connection per iteration, and a repeat.

Result: zero 0x00 / 22021 in any config, including plaintext. That last part matters - if the proxy were injecting a stray byte into the COPY stream, plaintext is exactly where it would show, because there's no TLS integrity check to tear the connection down first. It didn't show. So, on my instance/region at least, the evidence points away from byte-stream corruption as the cause.

The one thing I did see: a low rate (~0.1-1%) of connection drops mid-operation - 'server closed the connection unexpectedly' - across all configs, not correlated with connection age. Could be unrelated to your issue, but worth noting given the recent Metal-migration activity on the proxy layer.

What I'd suggest is to actually localize it on your side, since I couldn't from outside:

  • Run the same import over the private network (postgres.railway.internal from a service inside your project) and compare. If it stops failing there, the public proxy path is implicated, even if the mechanism isn't byte corruption.
  • Capture the exact bytes the server received for a failing row (e.g., log the raw COPY buffer your client emits, or round-trip via convert_to(col,'SQL_ASCII')) and diff against what you sent - that definitively separates 'our data' from 'something in transit'.
  • Note your client library + version; if you're not on psycopg3, the COPY framing on the wire differs, and that's worth testing.

chandrakiran-gr

I tried to reproduce this on a fresh Railway Postgres (Hobby) and couldn't trigger the 0x00 / 22021 error, which I think is itself a useful data point. Sharing the setup in case it helps you narrow it. What I ran: a Python/psycopg3 harness doing the same BEGIN -> COPY FROM STDIN (csv) into a temp staging table -> commit loop, over the public proxy host (*.proxy.rlwy.net). The generated data is asserted NUL-free before every send, so any 0x00 the server reported would have to have appeared after my buffer. ~2.6M rows total across four configs: TLS persistent connection, plaintext (sslmode=disable), fresh connection per iteration, and a repeat. Result: zero 0x00 / 22021 in any config, including plaintext. That last part matters - if the proxy were injecting a stray byte into the COPY stream, plaintext is exactly where it would show, because there's no TLS integrity check to tear the connection down first. It didn't show. So, on my instance/region at least, the evidence points away from byte-stream corruption as the cause. The one thing I did see: a low rate (~0.1-1%) of connection drops mid-operation - 'server closed the connection unexpectedly' - across all configs, not correlated with connection age. Could be unrelated to your issue, but worth noting given the recent Metal-migration activity on the proxy layer. What I'd suggest is to actually localize it on your side, since I couldn't from outside: - Run the same import over the private network (postgres.railway.internal from a service inside your project) and compare. If it stops failing there, the public proxy path is implicated, even if the mechanism isn't byte corruption. - Capture the exact bytes the server received for a failing row (e.g., log the raw COPY buffer your client emits, or round-trip via convert_to(col,'SQL_ASCII')) and diff against what you sent - that definitively separates 'our data' from 'something in transit'. - Note your client library + version; if you're not on psycopg3, the COPY framing on the wire differs, and that's worth testing.

alex-delia
PROOP

10 days ago

My setup is running bun with pg and pg-copy-streams. I had a similar issue using Insert From Unnest as well. Its just super odd its not occuring locally.


alex-delia

My setup is running bun with pg and pg-copy-streams. I had a similar issue using Insert From Unnest as well. Its just super odd its not occuring locally.

chandrakiran-gr
HOBBYTop 10% Contributor

10 days ago

That it also happens with INSERT FROM UNNEST is the key detail, and it points at Bun, not Railway.

UNNEST isn't a COPY stream, so the two failures share almost nothing except the pg driver encoding parameters and the runtime doing the socket writes. That rules out COPY-stream framing, and my sweep already rules out the proxy corrupting bytes. What's common is Bun. Bun reimplements Node's net/tls and Buffer internals, and that's a known rough spot, a multi-byte value or a buffer boundary can get mangled on the write path before it ever hits the wire. It shows up remotely because TLS plus larger/slower writes is exactly where that kind of bug surfaces, and stays hidden locally where writes are small and fast.

The test that settles it: run the exact same code under Node instead of Bun against the same remote DB. If Node is clean and Bun fails, it's Bun, full stop, and not Railway. Worth pinning your Bun version too and checking their issue tracker for pg/socket encoding bugs on your version.

If you want to stay on Bun meanwhile: for COPY, try text/CSV mode rather than binary; for UNNEST, add explicit casts (UNNEST($1::int[], $2::text[])) so the driver isn't leaning on Bun's binary encoding for ambiguous types. If those make it disappear, that's another finger pointing at the runtime's encoding path.


fedabalog46571-lab
FREE

10 days ago

Uploading…


alex-delia
PROOP

10 days ago

The main issue here is that this code worked for weeks without issue on the public database proxy, and has only started to cause issues in the last 2 days. Seems like something on Railway's end has changed for the issue to suddenly appear, or this would've been a problem from the start


Welcome!

Sign in to your Railway account to join the conversation.

Loading...