langchain_community.document_loaders.docusaurus
.DocusaurusLoader¶
- class langchain_community.document_loaders.docusaurus.DocusaurusLoader(url: str, custom_html_tags: Optional[List[str]] = None, **kwargs: Any)[source]¶
Load from Docusaurus Documentation.
It leverages the SitemapLoader to loop through the generated pages of a Docusaurus Documentation website and extracts the content by looking for specific HTML tags. By default, the parser searches for the main content of the Docusaurus page, which is normally the <article>. You can also define your own custom HTML tags by providing them as a list, for example: [“div”, “.main”, “a”].
Initialize DocusaurusLoader
- Parameters
url – The base URL of the Docusaurus website.
custom_html_tags – Optional custom html tags to extract content from pages.
kwargs – Additional args to extend the underlying SitemapLoader, for example: filter_urls, blocksize, meta_function, is_local, continue_on_failure
Attributes
web_path
Methods
__init__
(url[, custom_html_tags])Initialize DocusaurusLoader
aload
()Load text from the urls in web_path async into Documents.
fetch_all
(urls)Fetch all urls concurrently with rate limiting.
Lazy load text from the url(s) in web_path.
load
()Load sitemap.
load_and_split
([text_splitter])Load Documents and split into chunks.
parse_sitemap
(soup)Parse sitemap xml and load into a list of dicts.
scrape
([parser])Scrape data from webpage and return it in BeautifulSoup format.
scrape_all
(urls[, parser])Fetch all urls, then return soups for all results.
- __init__(url: str, custom_html_tags: Optional[List[str]] = None, **kwargs: Any)[source]¶
Initialize DocusaurusLoader
- Parameters
url – The base URL of the Docusaurus website.
custom_html_tags – Optional custom html tags to extract content from pages.
kwargs – Additional args to extend the underlying SitemapLoader, for example: filter_urls, blocksize, meta_function, is_local, continue_on_failure
- async fetch_all(urls: List[str]) Any ¶
Fetch all urls concurrently with rate limiting.
- load_and_split(text_splitter: Optional[TextSplitter] = None) List[Document] ¶
Load Documents and split into chunks. Chunks are returned as Documents.
- Parameters
text_splitter – TextSplitter instance to use for splitting documents. Defaults to RecursiveCharacterTextSplitter.
- Returns
List of Documents.
- parse_sitemap(soup: Any) List[dict] ¶
Parse sitemap xml and load into a list of dicts.
- Parameters
soup – BeautifulSoup object.
- Returns
List of dicts.
- scrape(parser: Optional[str] = None) Any ¶
Scrape data from webpage and return it in BeautifulSoup format.
- scrape_all(urls: List[str], parser: Optional[str] = None) List[Any] ¶
Fetch all urls, then return soups for all results.