langchain_community.document_loaders.concurrent
.ConcurrentLoader¶
- class langchain_community.document_loaders.concurrent.ConcurrentLoader(blob_loader: BlobLoader, blob_parser: BaseBlobParser, num_workers: int = 4)[source]¶
Load and pars Documents concurrently.
A generic document loader.
- Parameters
blob_loader – A blob loader which knows how to yield blobs
blob_parser – A blob parser which knows how to parse blobs into documents
Methods
__init__
(blob_loader, blob_parser[, num_workers])A generic document loader.
from_filesystem
(path, *[, glob, exclude, ...])Create a concurrent generic document loader using a filesystem blob loader.
get_parser
(**kwargs)Override this method to associate a default parser with the class.
Load documents lazily with concurrent parsing.
load
()Load all documents.
load_and_split
([text_splitter])Load all documents and split them into sentences.
- __init__(blob_loader: BlobLoader, blob_parser: BaseBlobParser, num_workers: int = 4) None [source]¶
A generic document loader.
- Parameters
blob_loader – A blob loader which knows how to yield blobs
blob_parser – A blob parser which knows how to parse blobs into documents
- classmethod from_filesystem(path: Union[str, Path], *, glob: str = '**/[!.]*', exclude: Sequence[str] = (), suffixes: Optional[Sequence[str]] = None, show_progress: bool = False, parser: Union[Literal['default'], BaseBlobParser] = 'default', num_workers: int = 4, parser_kwargs: Optional[dict] = None) ConcurrentLoader [source]¶
Create a concurrent generic document loader using a filesystem blob loader.
- Parameters
path – The path to the directory to load documents from.
glob – The glob pattern to use to find documents.
suffixes – The suffixes to use to filter documents. If None, all files matching the glob will be loaded.
exclude – A list of patterns to exclude from the loader.
show_progress – Whether to show a progress bar or not (requires tqdm). Proxies to the file system loader.
parser – A blob parser which knows how to parse blobs into documents
num_workers – Max number of concurrent workers to use.
parser_kwargs – Keyword arguments to pass to the parser.
- static get_parser(**kwargs: Any) BaseBlobParser ¶
Override this method to associate a default parser with the class.
- load_and_split(text_splitter: Optional[TextSplitter] = None) List[Document] ¶
Load all documents and split them into sentences.