[FEAT]: Auto-add and Watch Folder for Document Uploads

[Asentient]

What would you like to see?

Feature Request: Auto-add and Watch Folder for Document Uploads

Description:
Currently, users need to upload documents individually for the built-in RAG feature. It would be highly beneficial to have an option that allows the system to auto-add all documents from a specified folder and continuously watch the folder for new documents to be added automatically.

Benefits:

1. Streamlines the process of document management.
2. Saves time and reduces manual effort for users who deal with large volumes of documents.
3. Enhances the usability of the RAG feature, making it more efficient and user-friendly.

Proposed Solution:

• Implement a feature to specify a folder for automatic document uploads.
• The system should continuously monitor the folder and automatically add any new documents placed in it.

Use Case:
This feature would be particularly useful for businesses and researchers who frequently update their document repositories and need these documents to be readily available in AnythingLLM without manual intervention.

[thinkcodellc]

I also feel there is value in having API access to the endpoint where JSON payload can be used to add to the vector DB,

[PGTBoos]

Will it also handle document changes?. (ea after git pull or after a corection in a file ?.

[timothycarambat]

Will it also handle document changes?. (ea after git pull or after a corection in a file ?.

This feature is standalone already exists in beta, but you must manually select which file to sync. We do not auto-watch every document due to the probably case of someone wanting to auto-sync 10K files at the same time and that will for sure cause issues

[Asentient]

Will it also handle document changes?. (ea after git pull or after a corection in a file ?.

This feature is standalone already exists in beta, but you must manually select which file to sync. We do not auto-watch every document due to the probably case of someone wanting to auto-sync 10K files at the same time and that will for sure cause issues

Can we not at least select a folder to sync? Selecting a file does absolutely nothing for automating the process and would be quite unfortunate.

Other RAG systems do offer such options and I do not know if those used to having their RAG auto updating would bother with a system that requires every single file to be added manually.

Many applications would require such an option such as knowledge management systems, CRM systems, even generic daily note-taking etc. I do strongly believe that the auto update function would be a critical function to quite a few if not many. Just my two cents.

[PGTBoos]

As for windows file changes can trigger events, add to it a queue and for standalone it wouldn't matter even if people updated thousands docs which I rhough doubt people would do, but there are many people so perhaps. When will beta become prod?. I'm not on discord Outlook voor Android downloaden<https://aka.ms/ghei36>

…

________________________________ From: Asentient ***@***.***> Sent: Sunday, July 21, 2024 3:15:09 PM To: Mintplex-Labs/anything-llm ***@***.***> Cc: PGTBoos ***@***.***>; Manual ***@***.***> Subject: Re: [Mintplex-Labs/anything-llm] [FEAT]: Auto-add and Watch Folder for Document Uploads (Issue #1873) Will it also handle document changes?. (ea after git pull or after a corection in a file ?. This feature is standalone already exists in beta<https://docs.useanything.com/beta-preview/active-features/live-document-sync>, but you must manually select which file to sync. We do not auto-watch every document due to the probably case of someone wanting to auto-sync 10K files at the same time and that will for sure cause issues Can we not at least select a folder to sync? Selecting a file does absolutely nothing for automating the process and would be quite unfortunate. Other RAG systems do offer such options and I do not know if those used to having their RAG auto updating would bother with a system that requires every single file to be added manually. Many applications would require such an option such as knowledge management systems, CRM systems, even generic daily note-taking etc. I do strongly believe that the auto update function would be a critical function to quite a few if not many. Just my two cents. — Reply to this email directly, view it on GitHub<#1873 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AFQQOG34GXBSIW2IJV7R67LZNOX53AVCNFSM6AAAAABK5JKEOWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBRGYYDOMRZGI>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>