Add Session Management Feat.

[ajac-zero]

Has progress begun on llm session management? It would be an awesome feature.

Currently, our apps pull past conversation messages straight from our db upon each request, identified by a conversation id.

However, saving on-going conversations in-cache would be much faster and off-load some work from the db.

Is there an idea on how this would be implemented? Or should I pr something from scratch?

[justin-cechmanek]

It's under development and should be available soon.

[tylerhutcherson]

@ajac-zero work is available on this branch/PR
#141
Your feedback is welcomed.

[ajac-zero]

Awesome! The semantic session manager is really interesting.

I noticed each exchange is expected to be a pair of strings, would this limit the use of tool messages? Due to most providers returning tool/function calls as dicts. I suppose the simplest solution would be to stringify the response dicts before storing them, although this might require additional parsing. Alternatively using json mode would work fine already if available.

Also, there are situations where a single exchange could involve more than 2 messages. Like when a llm uses tools before generating a response.

user query message -> assistant tool call message -> tool result message -> assistant response message

For cases like these it could be useful to provide the option to store messages individually to allow greater flexibility.

[justin-cechmanek]

Hi @ajac-zero. The session managers do have a convenience store method to add a prompt and response texts in one call. This is meant for typical prompt->response->prompt->... chats. You can also call add_message to add messages individually in the form of dicts and these include tool call messages. Can you let us know if that meets your needs?