Cheshire3 Object Model - Tokenizer
==================================

.. _objects-tokenizer-api:

API
---

.. autoclass:: cheshire3.baseObjects.Tokenizer
   :members:


.. _objects-tokenizer-implementations:

Implementations
---------------

The following implementations are included in the distribution by default:

.. autoclass:: cheshire3.tokenizer.SimpleTokenizer

.. autoclass:: cheshire3.tokenizer.RegexpSubTokenizer

.. autoclass:: cheshire3.tokenizer.RegexpSplitTokenizer

.. autoclass:: cheshire3.tokenizer.RegexpFindTokenizer

.. autoclass:: cheshire3.tokenizer.RegexpFindOffsetTokenizer

.. autoclass:: cheshire3.tokenizer.RegexpFindPunctuationOffsetTokenizer

.. autoclass:: cheshire3.tokenizer.SentenceTokenizer

.. autoclass:: cheshire3.tokenizer.LineTokenizer

.. autoclass:: cheshire3.tokenizer.DateTokenizer

.. autoclass:: cheshire3.tokenizer.DateRangeTokenizer

.. autoclass:: cheshire3.tokenizer.PythonTokenizer
