Metadata-Version: 2.1
Name: tokenease
Version: 0.1.1
Summary: A simple and effective tokenizer.
License: MIT
Author: Mayank Nagda
Author-email: nagdamayank05@gmail.com
Requires-Python: >=3.10,<4.0
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Requires-Dist: scikit-learn (>=1.3.1,<2.0.0)
Requires-Dist: spacy (>=3.6.1,<4.0.0)
Description-Content-Type: text/markdown

# TokenEase
TokenEase is a versatile and efficient tokenizer, designed to streamline the process of converting text into tokens for natural language processing tasks. With its advanced features and customizable options, TokenEase provides a smooth and seamless experience for developers and researchers alike.

## Installation
Installation using pip:
    
```bash
pip install tokenease
```

Installation from source (easy to do if you want to contribute or modify the code!):

Ideally create a virtual environment for Python 3.10+ and install poetry. Then install tokenease with poetry:

```bash
poetry install
```

## Usage
(Coming soon...)
