Add Ollama

[jtpio]

Fixes #482

Ollama seems to be getting popular for running models locally, and looks like it would be good to have in Jupyter AI by default.

• Add OllamaProvider
• Add OllamaEmbeddingsProvider
• Expand the list of available models
• Add documentation
• Mark as experimental, like for GPT4All?

[jtpio]

I guess it would also be fine to have it as a custom model provider in a separate package, if we don't want to maintain it here in this repo.

[dlqqq]

Jeremy, thank you very much for working on this PR and for keeping up with the open source LLM ecosystem. I lack the time to engage there as much as I would like.

Let me know when this PR is ready, and I will approve & merge it once I run it locally. 👍

[lalanikarim]

Looking forward to this integration.
Is it feasible to have the list of supported models as configurable given that there are numerous finetunes out there that would be viable candidates with Ollama?

[jtpio]

Thanks @dlqqq and @lalanikarim!

Yeah I'll try to finish the PR soon. It was originally opened early to see if there was interest to have built-in support for Ollama in jupyter-ai.

Is it feasible to have the list of supported models as configurable given that there are numerous finetunes out there that would be viable candidates with Ollama?

Currently jupyter_ai seems to have support for allowing and blocking models:

jupyter-ai/packages/jupyter-ai/jupyter_ai/extension.py

Lines 61 to 75 in 642ac53

	allowed_models = List(
	Unicode(),
	default_value=None,
	help="""
	Language models to allow, as a list of global model IDs in the format
	`<provider>:<local-model-id>`. If `None`, all are allowed. Defaults to
	`None`.
	Note: Currently, if `allowed_providers` is also set, then this field is
	ignored. This is subject to change in a future non-major release. Using
	both traits is considered to be undefined behavior at this time.
	""",
	allow_none=True,
	config=True,
	)

Although not sure if that would be enough to use models that have not been added to the list of models in jupyter_ai.

[startakovsky]

🙇 @jtpio excited for this

[lalanikarim]

Since Ollama sports an OpenAI compatible Rest API, and jupyter-ai supports setting a base url to override the default OpenAI url, I created new local models based on models supported by Ollama and named them to match OpenAI models.

$ cat mistral-gpt-4.modelfile
FROM mistral

$ ollama create gpt-4 -f mistral-gpt-4.modelfile

This hack currently allows me to use Ollama with jupyter-ai.

Looking forward to integration to support local models hosted with Ollama.

[triinity2221]

@lalanikarim - I am also replicating a similar setup. Just curious are you able to use the /learn, /generate commands in the chat

[lalanikarim]

@lalanikarim - I am also replicating a similar setup. Just curious are you able to use the /learn, /generate commands in the chat

I haven't had any luck with /generate. I run into pydantic errors.

pydantic.v1.error_wrappers.ValidationError: 1 validation error for Outline
sections
  field required (type=value_error.missing)

I haven't tried /learn yet.

[lalanikarim]

@jtpio I am wondering if it would make sense to provide model name in a free form text field for Ollama models and to require %%ai magic code to include model name for ollama models and limit models to a predefined list.

%%ai ollama:mistral

Thoughts?

[jtpio]

@jtpio I am wondering if it would make sense to provide model name in a free form text field for Ollama models and to require %%ai magic code to include model name for ollama models and limit models to a predefined list.

Yeah I guess given the number of available models it would be very difficult to pick just a few here in the jupyter-ai package. And that would also require updating Jupyter AI regularly, increasing maintenance load.

So yes having a way to let users configure the list of models sounds like a good idea.

[siliconberry]

Since Ollama sports an OpenAI compatible Rest API, and jupyter-ai supports setting a base url to override the default OpenAI url, I created new local models based on models supported by Ollama and named them to match OpenAI models.

$ cat mistral-gpt-4.modelfile
FROM mistral

$ ollama create gpt-4 -f mistral-gpt-4.modelfile

This hack currently allows me to use Ollama with jupyter-ai.

Looking forward to integration to support local models hosted with Ollama.

Hi, Thanks for the hack. Its working. But curious.. how/what did you set the OPENAI API key ? Its working inside the chat box (Jupyternaut), but not inside the jupyter notebook. and its asking for OPEN_API_KEY. can u pl help ?

[Mrjaggu]

Anything update on this pr for adding ollama to jupyter ai..

[jtpio]

I'll try to finish this PR soon, and provide a way to configure the list of models (+ docs).

[dannongruver]

@jtpio: Thank you for this ollama integration Jeremy! Also look forward to configurable models. Let me know where I can buy you a coffee.

[lalanikarim]

Hi, Thanks for the hack. Its working. But curious.. how/what did you set the OPENAI API key ? Its working inside the chat box (Jupyternaut), but not inside the jupyter notebook. and its asking for OPEN_API_KEY. can u pl help ?

@siliconberry You can put any value for OPANAI_API_KEY. It is needed for this hack to work but will be ignored by ollama.

[dannongruver]

@lalanikarim in answer to @siliconberry question, have you gotten both the chat/jupyternaut as well as notebook cell (ie magic %%ai) working with ollama?

I’m struggling now to get notebook working. When I set the OPENAI_API_KEY environment variable to a dummy key (sk-abcd1234), the notebook gives an error indicating that ChatGPT doesn’t accept the key. Jupyternaut chat works fine. It seems like either jupyter_ai’s notebook is not using the config.json like Jupyternaut or ollama does not support all ChatGPT api…? Idk, maybe I’m missing something

[dannongruver]

Nevermind @lalanikarim @siliconberry. The magic %%ai does work.

Note: The magic doesn’t use the config.json but uses environment vars for key and url.

[cboettig]

Just joining the chorus of folks who would be excited to see this, especially given the rate at which new open-weights models are regularly appearing for ollama (llama3, phi3, wizardlm2). ollama's use of local GPU seems a lot smoother too than gpt4all too.

[orkutmuratyilmaz]

I'm so excited for this! 😍😍😍

[pedrogutobjj]

any updates?

[srdas]

@jtpio Please see also recent PR #868 in case this includes the remaining ideas you planned to implement?

[jtpio]

Thanks @srdas for the heads-up 👍

Looks like #868 lists more models, so we could continue with this PR instead of this one. But it would still be interesting for end users to have a way to configure the models, to avoid having to update Jupyter AI and make a new release each time there is a new model out there.

[srdas]

Thanks @srdas for the heads-up 👍

Looks like #868 lists more models, so we could continue with this PR instead of this one. But it would still be interesting for end users to have a way to configure the models, to avoid having to update Jupyter AI and make a new release each time there is a new model out there.

Thanks @jtpio -- let me test the new PR locally and also summarize what remains from what you have proposed. The drop down list is becoming really long so keeping on adding new models at the pace they are being included in Ollama may not be ideal both from a maintenance point of view and from an interface one. It may be preferable to handle Ollama in the way HuggingFace models are treated in the drop-down listing, which may address your note above.

[dlqqq]

@jtpio Thank you for checking in on this again! Since your PR does not duplicate the LangChain provider implementation and is older than #868, I think this PR (#646) should take precedence. Here are the next steps towards getting this merged:

• Rebase onto latest main, addressing conflicts
• Add additional model IDs listed in supporting local model for jupyter-ai with ollama #868 but not here. Please co-author @767472021 on the commit for contribution credit.
• Mark this PR as ready for review

Let me know if you need assistance!

[dlqqq]

But it would still be interesting for end users to have a way to configure the models, to avoid having to update Jupyter AI and make a new release each time there is a new model out there.

@jtpio I agree! This should probably be done in a future PR however, as it is a broader change that is not strictly necessary for Ollama support in Jupyter AI.

[pedrogutobjj]

Would it be possible to add Inline Completions models? Such as, https://www.ollama.com/library/deepseek-coder-v2, https://www.ollama.com/library/starcoder2???

[srdas]

@jtpio Awesome - the changes are working nicey. I tested it on the new embedding model list and it all seems to be working well.

[srdas]

Use Ollama capitalized in line 362 in "... to set up ollama ... "

Suggest adding a few more notes to the docs as follows:

Once Ollama is installed, download the model using the following command in a terminal:

ollama pull <model_name>

For example: ollama pull llama3. Do this for both LLM models and embedding models, as many as needed.

Once the models are downloaded, start up the Ollama server:

ollama serve

The models will then be available in jupyter-ai.

[3coins]

Good suggestion, but wouldn't block on this to merge. Also, this info can change based on ollama API, so we will be carrying instructions that are available in their docs and will have to update in case this changes.

[srdas]

@3coins Agree. I tested it again and it's all working so merging is predicated.

[srdas]

Thanks for adding the code to enable model choice.

[3coins]

@jtpio
Looks great!

[jtpio]

Thanks all for the reviews and the merge!

[pedrogutobjj]

I'm using the test version of 2.20, and I noticed that the line completion is not working, I tested it with the same model in VSCODE using the continue plugin and the code completion works, could it be a bug?

[srdas]

@pedrogutobjj - it is working at my end. Did you enable Inline completion in settings?

Example:

[pedrogutobjj]

my configs @srdas

[pedrogutobjj]

@srdas
i need use -instruct for completion model? Or can I use the same one?

Another question, if I enable the "Whether to fetch completions History provider." It doesn't provide suggestions for models, it provides suggestions for what I've used on other notebooks in the past.

[srdas]

You can use any model - I just used the instruct model as I could not find the version you are using on https://ollama.com/search. It shows this:

The run command gives a hint at what you need to name the model.
Also after you pull the Ollama model, you can run ollama list to see the exact name of the model.

I have to look into your second question some more.

[pedrogutobjj]

@srdas

[pedrogutobjj]

@srdas i need use -instruct for completion model? Or can I use the same one?

Another question, if I enable the "Whether to fetch completions History provider." It doesn't provide suggestions for models, it provides suggestions for what I've used on other notebooks in the past.

This happens when I activate the Whether to fetch completions History provider option, it brings codes that I have already used in the past, it has nothing to do with the model indications, as you mentioned in your post.

@srdas

[pedrogutobjj]

Another strange situation, even though there is no suggestion to complete the line, the gpu usage is at maximum, as if using a model, but nothing happens.