Local GPT compatible with text-generation-webui #8
meistaken
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi Pavel, thank you for your plugin. I'm pleased to share that your plugin works with the widely used AI frontend text-generation-webui, which is also compatible with Windows. Previously, I attempted to use the "Text Generator" plugin for Obsidian, but unfortunately, I couldn't get it to work with this frontend.
For anyone who might find this information useful a small guide how to get it works:
Start the text-generation-webui and select the desired model (in my case, it's openhermes-2.5-neural-chat-7b-v3-2-7b.Q5_K_M)
Configure a session
That's it. You can now access the Local GPT menu and see the results. I've tested it with a small 7b model (openhermes-2.5-neural-chat-7b-v3-2-7b.Q5_K_M), and on an RTX3090, I achieved 60.98 tokens/s. The quality is quite decent.
Beta Was this translation helpful? Give feedback.
All reactions