You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The current filtering feature under upload a document -> Data Connectors -> File Ignores, used for file uploads, has a few limitations and, for me, often results in unwanted documents being passed on to the LLM. The UI supports an ignore pattern, which technically can be inverted using ! to define what files to include. I mainly want to determine what files to include in the vector database rather than exclude, so I frequently use the inverse pattern.
The filtering feature is implemented by the ignore npm package and, from what I can tell, doesn't allow for nested inverts.
constignore=require('ignore');constfs=require('fs');constpath=require('path');constfiles=['README.md','index.js','docs/guide.md','src/components/Button.md','src/components/Button.js','src/utils/helpers.md','notes.txt',];constmatchFiles=(patterns)=>{constig=ignore().add(patterns);returnfiles.filter(file=>!ig.ignores(file));};constpatternsList=[['*','!.md'],['*','!*.md'],['*','!**/*.md'],['*','!.md','!**/*.md'],];patternsList.forEach((patterns,index)=>{console.log(`Test ${index+1} with patterns: ${patterns}`);constmatchedFiles=matchFiles(patterns);console.log('Matched files:',matchedFiles);console.log('---');});
This results in only the root files being included.
> node main.js
Test 1 with patterns: *,!.md
Matched files: []
---
Test 2 with patterns: *,!*.md
Matched files: [ 'README.md' ]
---
Test 3 with patterns: *,!**/*.md
Matched files: [ 'README.md' ]
---
Test 4 with patterns: *,!.md,!**/*.md
Matched files: [ 'README.md' ]
To simplify the UI, I suggest the following changes:
Allow the users to define files to be included using a glob pattern, i.e. **/**/*.md
Before sending the loaded data to the LLM, list all files matched by the pattern. This would allow us to test the pattern before sending it to the LLM.
What do you guys think? Would this be useful? I could look into a solution after we agree on an approach or if we decide to leave it be.
The text was updated successfully, but these errors were encountered:
What would you like to see?
The current filtering feature under
upload a document -> Data Connectors -> File Ignores
, used for file uploads, has a few limitations and, for me, often results in unwanted documents being passed on to the LLM. The UI supports an ignore pattern, which technically can be inverted using!
to define what files to include. I mainly want to determine what files to include in the vector database rather than exclude, so I frequently use the inverse pattern.The filtering feature is implemented by the
ignore
npm package and, from what I can tell, doesn't allow for nested inverts.This results in only the root files being included.
To simplify the UI, I suggest the following changes:
**/**/*.md
What do you guys think? Would this be useful? I could look into a solution after we agree on an approach or if we decide to leave it be.
The text was updated successfully, but these errors were encountered: