langchain_community.document_loaders.readthedocs.ReadTheDocsLoader¶

class langchain_community.document_loaders.readthedocs.ReadTheDocsLoader(path: Union[str, Path], encoding: Optional[str] = None, errors: Optional[str] = None, custom_html_tag: Optional[Tuple[str, dict]] = None, patterns: Sequence[str] = ('*.htm', '*.html'), exclude_links_ratio: float = 1.0, **kwargs: Optional[Any])[source]¶

Load ReadTheDocs documentation directory.

Initialize ReadTheDocsLoader

The loader loops over all files under path and extracts the actual content of the files by retrieving main html tags. Default main html tags include <main id=”main-content>, <div role=”main>, and <article role=”main”>. You can also define your own html tags by passing custom_html_tag, e.g. (“div”, “class=main”). The loader iterates html tags with the order of custom html tags (if exists) and default html tags. If any of the tags is not empty, the loop will break and retrieve the content out of that tag.

Parameters

  • path – The location of pulled readthedocs folder.

  • encoding – The encoding with which to open the documents.

  • errors – Specify how encoding and decoding errors are to be handled—this cannot be used in binary mode.

  • custom_html_tag – Optional custom html tag to retrieve the content from files.

  • patterns – The file patterns to load, passed to glob.rglob.

  • exclude_links_ratio – The ratio of links:content to exclude pages from. This is to reduce the frequency at which index pages make their way into retrieved results. Recommended: 0.5

  • kwargs – named arguments passed to bs4.BeautifulSoup.

Methods

__init__(path[, encoding, errors, ...])

Initialize ReadTheDocsLoader

lazy_load()

A lazy loader for Documents.

load()

Load documents.

load_and_split([text_splitter])

Load Documents and split into chunks.
__init__(path: Union[str, Path], encoding: Optional[str] = None, errors: Optional[str] = None, custom_html_tag: Optional[Tuple[str, dict]] = None, patterns: Sequence[str] = ('*.htm', '*.html'), exclude_links_ratio: float = 1.0, **kwargs: Optional[Any])[source]¶

Initialize ReadTheDocsLoader

The loader loops over all files under path and extracts the actual content of the files by retrieving main html tags. Default main html tags include <main id=”main-content>, <div role=”main>, and <article role=”main”>. You can also define your own html tags by passing custom_html_tag, e.g. (“div”, “class=main”). The loader iterates html tags with the order of custom html tags (if exists) and default html tags. If any of the tags is not empty, the loop will break and retrieve the content out of that tag.

Parameters

  • path – The location of pulled readthedocs folder.

  • encoding – The encoding with which to open the documents.

  • errors – Specify how encoding and decoding errors are to be handled—this cannot be used in binary mode.

  • custom_html_tag – Optional custom html tag to retrieve the content from files.

  • patterns – The file patterns to load, passed to glob.rglob.

  • exclude_links_ratio – The ratio of links:content to exclude pages from. This is to reduce the frequency at which index pages make their way into retrieved results. Recommended: 0.5

  • kwargs – named arguments passed to bs4.BeautifulSoup.

lazy_load() → Iterator[Document][source]¶

A lazy loader for Documents.

load() → List[Document][source]¶

Load documents.

load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶

Load Documents and split into chunks. Chunks are returned as Documents.

Parameters

text_splitter – TextSplitter instance to use for splitting documents. Defaults to RecursiveCharacterTextSplitter.

Returns

List of Documents.

Examples using ReadTheDocsLoader¶