[BUG]: Configuration to Enable/Disable LLM Streaming

[grundc]

What would you like to see?

I have an issue using AnythingLLM with our companies AzureOpenAI endpoint. When trying to start a chat the docker container is crashing and shutting down. In the log I can see the following error.

undefined
2024-07-03 16:20:18 /app/server/endpoints/chat.js:117
2024-07-03 16:20:18 error: e.message,
2024-07-03 16:20:18 ^
2024-07-03 16:20:18
2024-07-03 16:20:18 TypeError: Cannot read properties of undefined (reading 'message')
2024-07-03 16:20:18 at /app/server/endpoints/chat.js:117:20
2024-07-03 16:20:18 at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Not sure what exactly is the reason for it. But I got to notice that my company disabled "streaming" on the endpoint and something is telling me, that it migh be related to that.

I am wondering if there is a possibility to prevent anythingLLM from using the streaming approach.

[timothycarambat]

I didnt even know you could disable streaming for Azure endpoints. if that is the case - this is certainly the reason you are getting an error here

[grundc]

I didnt even know you could disable streaming for Azure endpoints. if that is the case - this is certainly the reason you are getting an error here

May be to clarify this. My company decided to put a proxy inbetween. So I am not consuming the Azure-Endpoint directly. So it might be that the endpoint handler has this disabled and not directly the Azure-Endpoint.

[timothycarambat]

This is almost certainly the culprit then. As an example, default nginx settings as a proxy will actually block streaming response and they will build in buffer and send as once completion