:mod:`langchain.text_splitter`.Tokenizer
===============================================

.. currentmodule:: langchain.text_splitter

.. autoclass:: Tokenizer

   
   
   .. rubric:: Attributes

   .. autosummary::
   
      ~Tokenizer.chunk_overlap
      ~Tokenizer.tokens_per_chunk
      ~Tokenizer.decode
      ~Tokenizer.encode
   
   

   
   
   .. rubric:: Methods

   .. autosummary::
   
      ~Tokenizer.__init__

   
   .. automethod:: Tokenizer.__init__

   
   


.. example_links:: Tokenizer