Server: add test for num slots, fails on master

[JohannesGaessler]

While working on #6828 I've been writing more tests to ensure that the results remain the same. However, while doing so I've noticed that for a given seed and a varying number of slots the results produced by the server are not deterministic. What I think is happening is that llama.cpp does not produce bit-for-bit identical results as the batch size is changed. Therefore, after some number of tokens two otherwise identical sequences randomly sample different tokens at which point they completely diverge.

I don't know if this can be fixed at all since the only way to get bit-for-bit identical results with floating point numbers is to do the exact same operations in the exact same order which would likely not yield good performance. Unless the CPU backend (which I used for testing) is supposed to produce bit-for-bit identical results in which case this would be indicative of a bug. In any case, feedback would be appreciated.

Sample outputs

content 0:  She was very proud of her story.
One day, she found an old cardboard box in her room. She was curious and decided to cut it in her project. She worked hard all day, cutting and tying until the box was ready.
Just then, her mom came into the room and said, "What are you doing, Sarah?"
Sarah smiled, "I'm cutting the box. I'm making a story about the truth!"
Her mom smiled and said, "That's very good. I'm so proud of you! Can I help you?"
Sarah nod
content 1:  She was very proud of her story.
One day, she found an old cardboard box in her room. She was curious and decided to cut it in her project. She worked hard all day, cutting and tying until the box was ready.
Just then, her mom came into the room and said, "What are you doing, Sarah?"
Sarah smiled, "I'm cutting the box. I'm making a story about the first one. Can you help me?"
Her mom smiled and said, "Of course! Let me see the surprise."
She took out a tube and showed

[JohannesGaessler]

I should mention, you run into this exact same problem if you have a fixed number of server slots but a varying number of parallel requests which I would argue is even more problematic.

[JohannesGaessler]

The sequences diverge for different batch sizes only if the temperature is high enough. I've added tests with temperature 0 and 1 and commented out those that currently fail on master. To assert that this is not a sampler issue I've expanded the tests around seeds: they now assert that the results are consistent with the same seed but different with different seeds.

I've changed the data type of context.seed. It is now a list of potentially different seeds (I think something like this was just not implemented). The interface for concurrent_requests now expects the user prompt as the first argument and the seed for the second argument (seed is not used for e.g. embeddings).

[compilade]

What I think is happening is that llama.cpp does not produce bit-for-bit identical results as the batch size is changed.

This is related to using a unified KV cache. See ggerganov/whisper.cpp#1941 (comment)

(I ran into this before in #6122 (comment))

[phymbert]

Minor: continuous batching is enabled by default (and cannot be disabled BTW :) )

[phymbert]

Thanks