I mean have you tried to run it on Railway yet? I have never had such issues on Railway myself.

I'm so sorry, I meant railway. Sheesh

The only header that Railway would touch is the server header.

The response header when I develop locally show that the response is chunked (streamed), but when deployed that chunked header is no longer present. I'm wondering, is there a configuration I can update to ensure my responses are streamed?

This would be a code issue as Railway will not modify such headers. Just did a test and was not able to see any headers disappear on me.

What specific headers are you setting in your application code?

I've attached a screen shot of the header in question from my local. Once deployed it no longer indicates chunked

I am unable to reproduce this, attached is a screenshot of a service running on Railway that has returned the correct headers.

I am having the same issue described here - @lorenseanstewart, were you able to resolve this?

To summarize:

• I have an Express server that I'm able to stream an LLM response from using Vercel's ai library (but only on my local machine)

• The frontend is a NextJS application running on Vercel

• I've verified it is not Vercel by running my localhost client and pointing it at my Railway service

• When I cURL the endpoint that should stream on localhost, it streams

• When I cURL the same endpoint once it's deployed, it is missing the Transfer-Encoding: chunked header

Is there potentially a proxy (nginx?) sitting in front of my service that is buffering the response? If so, can I adjust that behavior? Potentially related to the new "edge proxy"?

As I shown above I was not able to reproduce this on the legacy proxy, the original poster was also on the legacy proxy since the edge proxy was not an option 3 months ago.

My utilities app has since had the new edge proxy enabled, and I just tested again - https://utilities.up.railway.app/set-headers?Transfer-Encoding=chunked

Both curl and Postman show that the Transfer-Encoding was returned by the request, see attached screenshots.

So it is clear that this is a code issue, perhaps you are setting a content length along with the transfer encoding header?

Thanks for the quick reply, @brody!

I put together a quick demo to show what I'm seeing here in case it's helpful. Even when I strip this down to the bare minimum needed to stream a response, it's still not working on Railway (but does locally).

This is the repo

https://github.com/donholly/railway-streaming-test

You can hit the API via:

curl https://railway-streaming-test-production.up.railway.app -N -X POST -H 'Content-Type: text/plain' --data 'tell me a 200 word story about the moon'

These are the headers returned from cURL (using --verbose)

Note that there is a Content-Length header (that I am not setting) on this response and it's missing the Transfer-Encoding: chunked header...

> POST / HTTP/1.1
> Host: railway-streaming-test-production.up.railway.app
> User-Agent: curl/8.6.0
> Accept: */*
> Content-Type: text/plain
> Content-Length: 39
>
< HTTP/1.1 200 OK
< Content-Type: text/plain; charset=utf-8
< Date: Tue, 09 Jul 2024 17:57:48 GMT
< Server: railway-edge
< X-Custom-Header: hello
< X-Powered-By: Express
< X-Request-Id: v7T8MjzoR9Ko8Y5l2zaocQ_603909319
< Content-Length: 1224

When I run the exact same cURL locally via:

curl http://127.0.0.1:8080 -N -X POST -H 'Content-Type: text/plain' --data 'tell me a 200 word story about the moon'

It streams perfectly locally.

These are the headers from running locally.

Note that there is no content-length and Transfer-Encoding is correctly set to chunked

> POST / HTTP/1.1
> Host: 127.0.0.1:8080
> User-Agent: curl/8.6.0
> Accept: */*
> Content-Type: text/plain
> Content-Length: 46
>
< HTTP/1.1 200 OK
< X-Powered-By: Express
< Content-Type: text/plain; charset=utf-8
< x-custom-header: hello
< Date: Tue, 09 Jul 2024 17:59:37 GMT
< Connection: keep-alive
< Keep-Alive: timeout=5
< Transfer-Encoding: chunked

Let me know if you need anything else to help diagnose this and thank you for your help!

Note that there is a Content-Length header (that I am not setting)

Something in your code is setting it, please look into that.

Please see the attached screenshot showing SSE working with the curl command you provided.

I see that you are setting Content-Type to text/plain; charset=utf-8 try setting it to the correct value of text/event-stream; charset=UTF-8

Truly appreciate the followup here, @brody!

I have further diagnosed this and have come to the conclusion that it must be an issue with the corporate firewall/proxy on my machine.... Other devices of mine appear to be working correctly.

Strangely, other streaming (ChatGPT/Gemini) does seem to work on this machine, so it's not clear why only this implementation is affected... alas - not Railway's fault!

Hopefully this helps someone :)

Thanks again!

I would very much love to find out what the cause is, so please do let me know if / when you find out!

I am experiencing the same issue with the "Transfer-Encoding" header. It works locally but is not present in the response headers on my deployed app.

It seems that "Transfer-Encoding" is a hop-by-hop header (see here and here). As a result, it is "meaningful only for a single transport-level connection, and must not be retransmitted by proxies or cached."

When I just change the name of the header to "X-Transfer-Encoding", for example it appears in the headers of the deployed app as expected.

Could this be the issue? If so, any advice on how to handle it?

Please see my screenshots above, I was not able to reproduce the issue.

We would need a reproducible example to be able to look into your issue further.

Going to mark as solved for now though.