Poor performance for PDF QA

[enzolutions]

Hi @PromtEngineer

First, thank you for creating this public repository; this is a powerful and "simple" example of a local LLM.

I have used this project against the PDF https://assets.publishing.service.gov.uk/media/5f08a103e90e0712c5f221e0/UK_Points-Based_System_Further_Details_Web_Accessible.pdf from https://www.gov.uk/government/publications/uk-points-based-immigration-system-further-details-statement.

But when I try to ask simple questions like the type of visas or about the points system, the answers are erratic and delusional.

I did the vectorization using the model hkunlp/instructor-xl

I tried the LLM using

• MODEL_ID = "QuantFactory/Meta-Llama-3-8B-Instruct-GGUF"
• MODEL_BASENAME = "Meta-Llama-3-8B-Instruct.Q4_K_M.gguf"

And

• MODEL_ID = "AI-Engine/Meta-Llama-3-8B-Instruct-GGUF"
• MODEL_BASENAME = "Meta-Llama-3-8B-Instruct-imatrix.Q5_k_m.gguf"

With similar poor results.

So, I think maybe the problem is the vectorization or ChromaDB.

If anyone can try to use the PDF pointed out before, maybe you can suggest a new set of parameters or models to try to improve the quality of the QA.

Thanks in advance for any suggestions.