Metadata-Version: 2.1
Name: llmpool
Version: 0.1.3
Summary: Large Language Models' pool management library
Home-page: https://github.com/deep-diver/LLM-Pool
Author: chansung park
Author-email: deep.diver.csp@gmail.com
Keywords: LLM,instance pool,management
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: transformers
Requires-Dist: optimum
Requires-Dist: peft
Requires-Dist: text-generation
Requires-Dist: accelerate
Requires-Dist: bitsandbytes

# LLM-Pool

![](https://i.ibb.co/my2tf27/overview.png)

## Usecase

```python

from llmpool import LLModelPool
from llmpool import LocalLoRAModel
from llmpool import RemoteTxtGenIfLLModel

from transformers import AutoModelForCausalLM

model_pool = LLModelPool()
model_pool.add_model(
  # alpaca-lora 13b
  LocalLoRALLModel(
    "alpaca-lora-13b",
    "elinas/llama-13b-hf-transformers-4.29",
    "LLMs/Alpaca-LoRA-EvolInstruct-13B",
    model_cls=AutoModelForCausalLM
  ),
  
  RemoteTxtGenIfLLModel(
    "stable-vicuna-13b",
    "API_URL"
  )
)

for model in model_pool:
  batch_result = model.batch_gen(
    ["hello world"], 
    GenerationConfig(...)
  )
  
  _, stream_result = model.stream_gen(
    "hello world",
    GenerationConfig(...)
  )
  for text in stream_result:
    print(text, end='')

```

## Todo
- [ ] Add example notebooks
- [ ] Yaml parser to add models to model pool
