Events history cache clarifications

[baravit]

Hi guys, thanks for the latest updates!

We are building a chatbot with NeMo at the core.
Again, my humble kudos to you! In the past few months, we've been testing pretty much every tool out there suggested to help build secure LLM apps. Building our product with NeMo at the core was by far the most comfortable yet rewarding way to go :)

I'm trying to understand several things regarding the behavior of the event's history cache. Does the generate function expect to receive a history of all the messages thus far (how does it scale?)? Only the last n messages? Or just the latest one? I've noticed that it's an incremental process, regardless of how many messages are in the history. The app generates an event in response to the last user message and keeps accumulating them. This made me think that we should only send the last user message. However, in

NeMo-Guardrails/nemoguardrails/rails/llm/llmrails.py

Line 165 in 40889af

it seems like you are suggesting to keep accumulating the messages and send all of them to the generate function. When doing so, the keys of the cache chain all the user messages together along with the last message (delimited by ":"). This feels like it wasn't the intention.

What's the best way to approach this?

I also noticed this comment regarding the cache:

We keep a cache of the events history associated with a sequence of user messages.
TODO: when we update the interface to allow returning a "state object", this
should be removed.

Anything on what's coming next and how to prepare for that change?
Any production best practices estimation?

Again, thank you so much!
Bar.

[drazvan]

Thanks for the message @baravit! Let me try to provide more details on the design.

The guardrails interface is meant to be 100% compatible with the typical interface an LLM provides. This means a completion interface and a chat interface (think of the OpenAI endpoints for example; most LLM providers have something similar). This allows adding a guardrails configuration in front of an existing endpoint without any additional changes.

For the chat interface, you always send the chat history (and typically you cap it to the last N when it gets too big). The reason we have the cache is more subtle. The guardrails system operates in an "event-based" world, but the input and output are in a "message-based" world. In the event-based world, in addition to the events that correspond to the user and bot messages, we also have additional ones, e.g. the user intent, the bot intent, the execution of an action, the result of an action, etc. On the first call, you can create the events history from the message history (they are equivalent, just slightly different formatting). But on a second call, for example, you get [user_message_1, bot_message_1, user_message_2]. For the guardrails system to work well, we also need to know the other events between user_message_1 and bot_message_1. Without caching, we would have to compute them again, but computing them could be very expensive as it might involve additional LLM calls. And as you can imagine, in the 10th turn of the conversation it would no longer be feasible.

Coming back to the comment you've seen in the code, we also plan to add support for an explicit "state object" so that this caching will no longer be necessary. The state will be managed explicitly by the developer. Most likely, the interface will be something along the lines new_state = llm_rails.generate_state(new_user_message, current_state). And the state object will have a messages array with the conversation and an events array with the expanded set of events. We'll also add support to submit any type of event through this interface, e.g. new_state = llm_rails.generate_state(new_event, current_state). I know this is a bit abstract, happy to provide some more examples if needed. In terms of timeline, I believe that in the next 3 months we'll add this interface as well.

[baravit]

Hi @drazvan!

I wanted to follow up here and point out that when sending Nemo last N messages, that for some reason (i.e server restart..) doesn't have cached events, the flow completely breaks.

you can see the following example:

import asyncio
import os
from nemoguardrails import LLMRails, RailsConfig
from dotenv import load_dotenv

import logging
logLevel = logging.INFO
logging.basicConfig(level=logLevel)

load_dotenv()

YAML_CONFIG = """
models:
  - type: main
    engine: openai
    model: text-davinci-003
"""
COLANG_CONFIG = """
define user express greeting
  "hey"
  "yo"
  "hi"
  "hello"
  "what's up"

define bot express greeting
  "Hello $user.name!"

define flow
  user express greeting
  bot express greeting

  
define user ask test
  "test test test?"

define bot inform test
  "test test test!"

define bot inform not test
  "NOT!"

define flow test
  user ask test
  $bool = True
  if $bool
    bot inform test
  else
    bot inform not test
    
"""

config = RailsConfig.from_content(COLANG_CONFIG, YAML_CONFIG)
APP = LLMRails(config, verbose=True)

async def get_message(history):
    try:
        bot_message = await APP.generate_async(messages=history)
        return bot_message
    except Exception as e:
        print(e) 

def main():
    history = [
        {
            "role": "context", 
            "content": {
                "user": {
                  "name": "Bar"
                }
            }
        },
        {"role": "user", "content": "Hi"},
        {"role": "assistant", "content": "Hi Leon, How can I help you today 🙂?"},
        {"role": "user", "content": "What's up?"},
        {"role": "assistant", "content": "I'm doing great! What can I do for you today?"},
        {"role": "user", "content": "I'm great!"},
        {"role": "assistant", "content": "So, what's up?"},
        {"role": "user", "content": "What's the language in Haiti?"},
        {"role": "assistant", "content": "The official language of Haiti is French, however, Haitian Creole is also widely used."},
        {"role": "user", "content": "And where is it located?"},
        {"role": "assistant", "content": "Haiti is located in the Caribbean Sea, between the Dominican Republic and Cuba."},
        {"role": "user", "content": "test test test?"},
    ]
    result = asyncio.run(get_message(history))
    print(result)

if __name__ == "__main__":
    main()

From debug logs, you can see how things get messy in generate next step when building the current conversation between the user and the bot.

Again, would love to help in any way I can,
And thank you for your work here!

Bar.

[d-shree]

@drazvan - Is this new_state = llm_rails.generate_state(new_user_message, current_state) now supported?

I have a requirement where while generating the user intent (i.e., generate_user_intent) , I'd like the user_intent to be generated based on some context/history of messages, and since I'm using the Python API, I'm unsure how to go about this.

Could you suggest the best way to do this?

[maziyarpanahi]

I am also interested in this discussion as I am planning to use NeMo in a project with a stateless LLM. Similar to the LLM itself, I am not planning to keep anything in the memory/db/etc regarding the history. The history is the list of messages sent by the users similar to the OpenAI. (even with LangChain, I construct the chat_history myself and feed it alongside the input)

Would it be possible to turn off this caching? And if yes, how heavy/time-consuming would it be for a history of 8 to 10 messages? and how would this cache part scale if you have 10k users?

[maziyarpanahi]

Hi @drazvan
Any update on stateless rails so developers can manage it instead of it being cached internally?

[drazvan]

Thanks for bumping up this discussion @maziyarpanahi. Let's work together to come up with the best solution.

There are a few ways we can go about this:

1. We also return a "state object" as part of the API. What's in there shouldn't matter to the user; it just needs to be provided back when the next message needs to be processed. This is a relatively easy change, it essentially means returning what is being now cached.
2. We allow the option to compute the canonical forms for all the messages in the history. This will basically re-play the conversation from the beginning, involving the LLM whenever needed. While possible, this is not practical as it will add a huge latency in subsequent turns.
3. We make the core logic work by computing only the canonical form for the current user message. There are some changes we need to make to the prompt. However, this would not be equivalent to option 1. For example, if we are at turn 5 in the conversation, and turn 3 would have started a flow because of the intent in that turn, then in this approach, that flow would not be started (because we did not compute the user intent in turn 3).

The bottom line would be that we need to process each turn individually to have strong guardrails using dialog flows (which need to compute the user intent at every step).

Which option would be the best fit for your use case?

[maziyarpanahi]

Hi @drazvan

Thank you so much for putting the time and explaining this process so clearly. My thoughts:

1. This option which seems to exist is actually very useful to me to build a stateless LLM service. I can keep track of this state object alongside with the chat_history. For a UI scenario this can happen behind the scene and saved next to the conversations, and for an API scenario, this can be saved/retrieved based on conversation_id without the user's knowledge.
2. This seems to be interesting as a fallback just in case the workflow I described in 1. fails to retrieve any state object or it was not a correct object, then falls back to this one-time only operation which seems expensive but necessary to prepare everything to go back to the best practice in 1.. (I would also say, if we introduce this in a Chatbot that already exist, old conversations can enjoy this option, the UI may take a few seconds for the first time, but then we are ready for 1.

Would it be possible to have a quick example to show how to do options 1 and 2 both with and without other integrations like LangChain (if that matters)? With these two options, it will be super easy to implement the desire guardrails for any LLM models as long as we keep track of the required state object and in case of catastrophe, falling back on a one-time reconstructing the whole state object.

[drazvan]

The way you can achieve this right now, without any changes, is to manually extract the entry from the cache (the one saved here: https://github.com/NVIDIA/NeMo-Guardrails/blob/develop/nemoguardrails/rails/llm/llmrails.py#L452), right after the generate/generate_async call. Something along these lines (not tested!):

messages = [...]

# Retrieve the `state` from somewhere and add it to the cache
cache_key = get_history_cache_key(messages)
rails.events_history_cache[cache_key] = state

# Generate the new message
new_message = await rails.generate_async(messages=messages)

# Extract the `state`
cache_key = get_history_cache_key(messages + [new_message])
state = rails.events_history_cache[cache_key]

# Save the state somewhere

That being said, we'll add a proper state interface in the coming months, likely end of March. And yes, the fallback mechanism is an interesting one to support, to re-generate in case the information got lost.