SetMemory() not available for ollama.

[MukulManni]

I tried to use gollm.SetMemory(4000) with ollama. But it gave, no encoding for model llama3-un error. (llama3-un is the name of model)

Code:

llm, err := gollm.NewLLM(
		gollm.SetProvider("ollama"),
		gollm.SetModel("llama3-un"),
		gollm.SetDebugLevel(gollm.LogLevelWarn),
		gollm.SetMemory(400),
	)

Output:

2024/08/17 14:08:18 ERROR: Failed to create LLM with memory [error failed to get encoding for model: no encoding for model llama3-un]
2024/08/17 14:08:18 Failed to create LLM: failed to create LLM with memory: failed to get encoding for model: no encoding for model llama3-un
exit status 1

I searched the docs and found that some kind of encoding is being used to make memory and the encodings are only available for openai models.

Can there be a way to use memory in ollama models.

[MukulManni]

I think to add memory in a models, we have to know it's prompt, which is different for different models.
So to use memory in open source models, we have to provide the prompts on which the model was trained.
In gollm if we somehow add an option to provide prompt, then we can maybe add memory in open source models like in ollama,
or otherwise simply write code with prompts to use memory in the model.

[teilomillet]

Thank you MukulManni,

The error likely comes from the internal/memory.go, for the NewMemory function.

I suspect it's is because we use github.com/pkoukk/tiktoken-go under the hood and it's a tokenizer for openai models, then logically not recognizing llama3.1.

I guess we could default to a gpt-4o models in the case of an error, this would still be better than nothing, except if you can find a tokenizer for llama3.1 models.

The change looks like :

func NewMemory(maxTokens int, model string, logger Logger) (*Memory, error) {
encoding, err := tiktoken.EncodingForModel(model)
if err != nil {
logger.Warn("Failed to get encoding for model, defaulting to gpt-4o", "model", model, "error", err)
encoding, err = tiktoken.EncodingForModel("gpt-4o")
if err != nil {
return nil, fmt.Errorf("failed to get default encoding: %v", err)
}
}

return &Memory{
	messages:  []Message{},
	maxTokens: maxTokens,
	encoding:  encoding,
	logger:    logger,
}, nil

}

---

I just pushed the change, can you confirm it's working on your side as well ?