langchain[minor]: Multi-file loader #5584
Conversation
dosubot
bot
added
the
size:L
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
dosubot
bot
added
the
auto:enhancement
| import { Document } from "@langchain/core/documents"; | ||
| import { getEnv } from "@langchain/core/utils/env"; | ||
| import { BaseDocumentLoader } from "../base.js"; | ||
| import { type LoadersMapping, UnknownHandling } from "./directory.js"; |
There was a problem hiding this comment.
I'm not sure if these imports should be refactored / moved to some common file instead of being imported from directory.js.
Open to suggestions.
| * | ||
| * ``` | ||
| */ | ||
| export class MultiFileLoader extends BaseDocumentLoader { |
There was a problem hiding this comment.
This implementation shares a lot in common with DirectoryLoader that I wonder if this is something that can be moved to a new base class so there is reuse.
| * indicating that the modules failed to load. | ||
| * @returns A promise that resolves to an object containing the imported functions. | ||
| */ | ||
| static async imports(): Promise<{ |
There was a problem hiding this comment.
We don't have to do this anymore, you can just import them at the top as normal:
https://github.com/langchain-ai/langchainjs/blob/main/.github/contributing/INTEGRATIONS.md
There was a problem hiding this comment.
This was pretty much copied from the directory loader. Should I also remove it from there as well?
I can update the config file accordingly as mentioned in the doc you posted.
|
Thanks for the PR! Is this materially different from calling the |
|
It's a completely different use-case. I have files from different directories that I may pick and choose from vs entire directories, so looping with the DirectoryLoader wouldn't work as that is on a directory-basis. I'm trying to build something for myself where I feed in source code files that maybe have imports from different paths so I don't need a full directory read. |
jacoblee93
added
lgtm
|
Thanks! Sorry for delay |
Introduces the
MultiFileLoader, which behaves like theDirectoryLoader, except the input is a list of file paths instead of a single directory path.