langchain.text_splitter
.split_text_on_tokens¶
- langchain.text_splitter.split_text_on_tokens(*, text: str, tokenizer: Tokenizer) List[str] [source]¶
Split incoming text and return chunks using tokenizer.