|  | # Cheshire3 Objects: Tokenizer

## Description

A Tokenizer takes a string of language and processes it to produce an ordered list of tokens.

## Implementations

The following implementations are pre-configured and ready to use.
They may be used out of the box in configurations for Indexes, Workflows etc.

- SimpleTokenizer
  A basic Tokenizer that splits the given data at whitespace.
- RegexpSubTokenizer
  A Tokenizer that replaces regular expression matches in the data with a configurable character (defaults to whitespace), then splits the result at whitespace.
- RegexpSplitTokenizer
  A Tokenizer that splits the data at matches of a regular expression. Each match (e.g. punctuation) is preserved in the result as a separate token.
- RegexpFindTokenizer
  A Tokenizer that uses a regular expression to finds tokens. Matches a large range commonly occurring tokens (e.g. simple words, email@addresses, £monetary $amounts, ti:me:s, percentages%, a.c.r.o.n.y.m.s., hyphen-ated and abbrev'd words etc.)
- RegexpFindOffsetTokenizer
  A Tokenizer that uses a regular expression to finds tokens in the same way as RegexpFindTokenizer, but also maintains the character offset where token occurred within the data (useful for proximity searching within N characters and search term highlighting.)
- SentenceTokenizer
  A Tokenizer to split data into sentences (i.e. splits at full-stops.)
- PythonTokenizer
  A Tokenizer to split and annotate tokens in the Python programming language.
- DateTokenizer
  A Tokenizer that finds date time tokens within the given data.
- PreserveMaskingTokenizer
  A keyword Tokenizer based on RegexpSubTokenizer but which doesn't split at CQL masking characters (^*?). Especially useful when tokenizing incoming queries.

## API

Module: cheshire3.tokenizer
Classes:

- SimpleTokenizer
  Splits the given data at all instances of a configurable string. Defaults to splitting on whitespace.
- OffsetTokenizer
  Abstract base class for Tokenizers that maintain information about the location of each token as character offsets.
- RegexpSubTokenizer
  Replaces regular expression matches in the data with a configurable character (defaults to whitespace), then splits the result at whitespace.
- RegexpSplitTokenizer
  Splits the data at matches of a regular expression. By default, each match (e.g. punctuation) is preserved in the result as a separate token.
- RegexpFindTokenizer
  Returns all words that match the configured regex.
- RegexpFindOffsetTokenizer
  Uses a regular expression to finds tokens in the same way as RegexpFindTokenizer, but also maintains the character offset where token occurred within the data.
- RegexpFindPunctuationOffsetTokenizer As RegexpFindOffsetTokenizer but does not include punctuation when calculating offsets.
- LineTokenizer
  Splits the data at new line characters
- SentenceTokenizer
  Splits data into sentences (i.e. splits at full-stops.)
- PythonTokenizer
  Splits and annotates tokens in the Python programming language.
- DateTokenizer
  Finds date time tokens within the given data.

Methods:

Function | Parameters | Returns | Description
--- | --- | --- | ---
__init__ | config, parent |  |
process_string | session, data | list | Process a raw string to produce an ordered list of tokens
process_hash | session, data | list | Process the text of each item in the hash to produce an ordered list of tokens for that hash entry.

### Sub-Packages