![](https://private-user-images.githubusercontent.com/100853494/312098218-cebecf35-bdd6-459c-8a1e-bad7c9a8037b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQzNzI3NTgsIm5iZiI6MTcyNDM3MjQ1OCwicGF0aCI6Ii8xMDA4NTM0OTQvMzEyMDk4MjE4LWNlYmVjZjM1LWJkZDYtNDU5Yy04YTFlLWJhZDdjOWE4MDM3Yi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODIzJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyM1QwMDIwNThaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT04NTA0ZDM3ZmE0YTE0YzA3NGU1ZWYzOWQzMGE1NWU3N2JkZGM2YmYyY2E0Y2I0NjFkNWIxYTQ2MzQ3Y2M0ZWZkJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.ZLig5Pqq7qqPEim6Fc9F9HIUhxtEq77mLWHbyPjKxuM)
This project introduces a novel approach to Cosmos data analysis, diverging from conventional techniques reliant on embeddings and vector databases. The methodology employed involves a TFID retriever, complemented by a meticulous long context reordering and a flash reranker. This innovative approach results in a substantial improvement in retrieval speed, particularly noteworthy with a parameter setting of k=8, while concurrently achieving accuracy levels closely aligned with those obtained through vector databases.
S.No | RAG with VectorDB | RAG without VectorDB | |
---|---|---|---|
1. | Methodology | Vector Embeddings | TFID Retriever + LC Reorder + FlashReRanker |
2. | Storage Requirements | Higher storage demand with additional computation | Minimal storage needed solely for the data |
3. | Retrieval Speed | Rapid retrieval with VectorDB | Moderately fast retrieval without VectorDB |
4. | Retrieval Accuracy | Achieves k <= 5 | Attains k <= 8 |
![](https://private-user-images.githubusercontent.com/100853494/312141129-e2265159-9ec7-4c88-85c9-a2f18ccf14ef.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQzNzI3NTgsIm5iZiI6MTcyNDM3MjQ1OCwicGF0aCI6Ii8xMDA4NTM0OTQvMzEyMTQxMTI5LWUyMjY1MTU5LTllYzctNGM4OC04NWM5LWEyZjE4Y2NmMTRlZi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODIzJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyM1QwMDIwNThaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1kYjA2NWU0MzY3MmQ2NTNjMDU1ZjJmOTM1OTAyYWEyOWVkMWZkNzkwZTI3MmVkN2FhNmYyNjI5MjViM2RiNDYxJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.ZTXXa-ksaMIb1tdMWvct38mv4F336S-IU1GYCmfpU5g)
Special thanks to Prithiviraj Damodaran for developing a light-weight and powerful re-ranker.