langchain_community.document_loaders.docusaurus.DocusaurusLoader

class langchain_community.document_loaders.docusaurus.DocusaurusLoader(url: str, custom_html_tags: Optional[List[str]] = None, **kwargs: Any)

Load from Docusaurus Documentation.

It leverages the SitemapLoader to loop through the generated pages of a Docusaurus Documentation website and extracts the content by looking for specific HTML tags. By default, the parser searches for the main content of the Docusaurus page, which is normally the <article>. You can also define your own custom HTML tags by providing them as a list, for example: [“div”, “.main”, “a”].

Initialize DocusaurusLoader

Parameters

• url (str) – The base URL of the Docusaurus website.

• custom_html_tags (Optional[List[str]]) – Optional custom html tags to extract content from pages.

• kwargs (Any) – Additional args to extend the underlying SitemapLoader, for example: filter_urls, blocksize, meta_function, is_local, continue_on_failure

Attributes

web_path

Methods

__init__(url[, custom_html_tags])	Initialize DocusaurusLoader
alazy_load()	A lazy loader for Documents.
aload()	Load text from the urls in web_path async into Documents.
fetch_all(urls)	Fetch all urls concurrently with rate limiting.
lazy_load()	Load sitemap.
load()	Load data into Document objects.
load_and_split([text_splitter])	Load Documents and split into chunks.
parse_sitemap(soup)	Parse sitemap xml and load into a list of dicts.
scrape([parser])	Scrape data from webpage and return it in BeautifulSoup format.
scrape_all(urls[, parser])	Fetch all urls, then return soups for all results.

__init__(url: str, custom_html_tags: Optional[List[str]] = None, **kwargs: Any)

Initialize DocusaurusLoader

Parameters

• url (str) – The base URL of the Docusaurus website.

• custom_html_tags (Optional[List[str]]) – Optional custom html tags to extract content from pages.

• kwargs (Any) – Additional args to extend the underlying SitemapLoader, for example: filter_urls, blocksize, meta_function, is_local, continue_on_failure

async alazy_load() → AsyncIterator[Document]

A lazy loader for Documents.

Return type

AsyncIterator[Document]

aload() → List[Document]

Load text from the urls in web_path async into Documents.

Return type

List[Document]

async fetch_all(urls: List[str]) → Any

Fetch all urls concurrently with rate limiting.

Parameters

urls (List[str]) –

Return type

Any

lazy_load() → Iterator[Document]

Load sitemap.

Return type

Iterator[Document]

load() → List[Document]

Load data into Document objects.

Return type

List[Document]

load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]

Load Documents and split into chunks. Chunks are returned as Documents.

Do not override this method. It should be considered to be deprecated!

Parameters

text_splitter (Optional[TextSplitter]) – TextSplitter instance to use for splitting documents. Defaults to RecursiveCharacterTextSplitter.

Returns

List of Documents.

Return type

List[Document]

parse_sitemap(soup: Any) → List[dict]

Parse sitemap xml and load into a list of dicts.

Parameters

soup (Any) – BeautifulSoup object.

Returns

List of dicts.

Return type

List[dict]

scrape(parser: Optional[str] = None) → Any

Scrape data from webpage and return it in BeautifulSoup format.

Parameters

parser (Optional[str]) –

Return type

Any

scrape_all(urls: List[str], parser: Optional[str] = None) → List[Any]

Fetch all urls, then return soups for all results.

Parameters

• urls (List[str]) –

• parser (Optional[str]) –

Return type

List[Any]