langchain.text_splitter.split_text_on_tokens¶

langchain.text_splitter.split_text_on_tokens(*, text: str, tokenizer: Tokenizer) → List[str][source]¶

Split incoming text and return chunks using tokenizer.