---
icon: lucide/mic
---

# whisper

Run a local ASR server with automatic backend selection based on your platform:

- **macOS Apple Silicon** → [mlx-whisper](https://github.com/ml-explore/mlx-examples/tree/main/whisper) (Metal acceleration)
- **Linux/CUDA** → [faster-whisper](https://github.com/SYSTRAN/faster-whisper) (CTranslate2)
- **HuggingFace** → [transformers](https://huggingface.co/docs/transformers/model_doc/whisper) (supports safetensors models and known remote-code ASR models such as Cohere Transcribe)

> [!NOTE]
> **Quick Start** - Get transcription working in 30 seconds:
> ```bash
> pip install "agent-cli[faster-whisper]"
> agent-cli server whisper
> ```
> Server is now running at `http://localhost:10301`. Verify with `curl http://localhost:10301/health`.
>
> Apple Silicon MLX-only setup:
> ```bash
> pip install "agent-cli[mlx-whisper]"
> agent-cli server whisper --backend mlx
> ```
>
> Cohere Transcribe via HuggingFace transformers:
> ```bash
> pip install "agent-cli[whisper-transformers]" librosa soundfile sentencepiece protobuf
> agent-cli server whisper \
>   --backend transformers \
>   --model CohereLabs/cohere-transcribe-03-2026 \
>   --default-language en
> ```
>
> Use it with any OpenAI-compatible client, or configure agent-cli to use it - see [Configuration](../../configuration.md#using-local-whisper-server).

## Features

- **OpenAI-compatible API** at `/v1/audio/transcriptions` - drop-in replacement for OpenAI's Whisper API
- **Wyoming protocol** for [Home Assistant](https://www.home-assistant.io/) voice integration (Wyoming is the standard protocol for local voice services)
- **TTL-based memory management** - models unload after idle period, freeing RAM/VRAM
- **Multiple models** - run different model sizes with independent TTLs
- **Background preloading** - downloads start at startup without blocking; use `--preload` to wait
- **Multi-platform support** - automatically uses the optimal backend for your hardware

> [!IMPORTANT]
> Some backend/model combinations support fewer features than Whisper.
> For example, Cohere Transcribe currently requires an explicit language and does not provide translation or timestamped subtitle output.

## Usage

```bash
agent-cli server whisper [OPTIONS]
```

## Examples

```bash
# Run with default large-v3 model (5-minute TTL)
agent-cli server whisper

# Use smaller model with 10-minute TTL
agent-cli server whisper --model small --ttl 600

# Run multiple models (requests can specify which to use)
agent-cli server whisper --model large-v3 --model small --default-model large-v3

# Force CPU mode
agent-cli server whisper --device cpu

# Download model without starting server (requires faster-whisper)
agent-cli server whisper --model large-v3 --download-only

# Preload model at startup and wait until ready
agent-cli server whisper --preload

# Run Cohere Transcribe through the transformers backend
agent-cli server whisper \
  --backend transformers \
  --model CohereLabs/cohere-transcribe-03-2026 \
  --default-language en
```

## Options

<!-- CODE:START -->
<!-- from agent_cli.docs_gen import all_options_for_docs -->
<!-- print(all_options_for_docs("server.whisper")) -->
<!-- CODE:END -->
<!-- OUTPUT:START -->
<!-- ⚠️ This content is auto-generated by `markdown-code-runner`. -->
### Options

| Option | Default | Description |
|--------|---------|-------------|
| `--model, -m` | - | Whisper model(s) to load. Common models: `tiny`, `base`, `small`, `medium`, `large-v3`, `distil-large-v3`. Can specify multiple for different accuracy/speed tradeoffs. Default: `large-v3` |
| `--default-model` | - | Model to use when client doesn't specify one. Must be in the `--model` list |
| `--device, -d` | `auto` | Compute device: `auto` (detect GPU), `cuda`, `cuda:0`, `cpu`. MLX backend always uses Apple Silicon |
| `--compute-type` | `auto` | Precision for faster-whisper: `auto`, `float16`, `int8`, `int8_float16`. Lower precision = faster + less VRAM |
| `--cache-dir` | - | Custom directory for downloaded models (default: HuggingFace cache) |
| `--default-language` | - | Fallback language code for requests that omit `language`. Required for models that do not support language auto-detection (for example Cohere Transcribe). |
| `--trust-remote-code` | `false` | Allow Hugging Face model repositories to execute custom Python code. Known supported remote-code ASR models are trusted automatically. |
| `--ttl` | `300` | Seconds of inactivity before unloading model from memory. Set to 0 to keep loaded indefinitely |
| `--preload` | `false` | Load model(s) immediately at startup instead of on first request. Useful for reducing first-request latency |
| `--host` | `0.0.0.0` | Network interface to bind. Use `0.0.0.0` for all interfaces |
| `--port, -p` | `10301` | Port for OpenAI-compatible HTTP API (`/v1/audio/transcriptions`) |
| `--wyoming-port` | `10300` | Port for Wyoming protocol (Home Assistant integration) |
| `--no-wyoming` | `false` | Disable Wyoming protocol server (only run HTTP API) |
| `--download-only` | `false` | Download model(s) to cache and exit. Useful for Docker builds |
| `--backend, -b` | `auto` | Inference backend: `auto` (faster-whisper on CUDA/CPU, MLX on Apple Silicon), `faster-whisper`, `mlx`, `transformers` (HuggingFace, supports safetensors and known remote-code ASR models) |

### General Options

| Option | Default | Description |
|--------|---------|-------------|
| `--log-level` | `info` | Set logging level. |


<!-- OUTPUT:END -->

## API Endpoints

Once running, the server exposes:

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/v1/audio/transcriptions` | POST | OpenAI-compatible transcription |
| `/v1/audio/translations` | POST | OpenAI-compatible translation (to English) |
| `/v1/audio/transcriptions/stream` | WebSocket | Real-time streaming transcription |
| `/v1/model/unload` | POST | Manually unload a model from memory |
| `/health` | GET | Health check with model status |
| `/docs` | GET | Interactive API documentation |

> [!NOTE]
> `/v1/audio/translations`, `srt`, and `vtt` output depend on the selected backend/model.
> Whisper models with timestamp-capable backends support them; Cohere Transcribe and the generic transformers backend do not provide timestamped subtitle output.

## Using the API

### curl Example

```bash
# Transcribe an audio file
curl -X POST http://localhost:10301/v1/audio/transcriptions \
  -F "file=@recording.wav" \
  -F "model=whisper-1"

# With language hint and verbose output
curl -X POST http://localhost:10301/v1/audio/transcriptions \
  -F "file=@recording.wav" \
  -F "model=whisper-1" \
  -F "language=en" \
  -F "response_format=verbose_json"

# Get SRT subtitles
curl -X POST http://localhost:10301/v1/audio/transcriptions \
  -F "file=@recording.wav" \
  -F "model=whisper-1" \
  -F "response_format=srt"
```

### Python Example (OpenAI SDK)

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:10301/v1", api_key="not-needed")

# Transcribe audio
with open("recording.wav", "rb") as f:
    transcript = client.audio.transcriptions.create(
        model="whisper-1",
        file=f,
    )
print(transcript.text)
```

## WebSocket Streaming Protocol

The `/v1/audio/transcriptions/stream` endpoint provides real-time streaming transcription.

### Protocol

1. Connect to `ws://localhost:10301/v1/audio/transcriptions/stream?model=whisper-1`
2. Send binary audio chunks (16kHz, 16-bit, mono PCM)
3. Send `EOS` (3 bytes: `0x45 0x4F 0x53`) to signal end of audio
4. Receive JSON response with transcription

### Message Format

**Server response:**
```json
{
  "type": "final",
  "text": "transcribed text here",
  "is_final": true,
  "language": "en",
  "duration": 3.5,
  "segments": [...]
}
```

**Error response:**
```json
{"type": "error", "message": "error description"}
```

## Model Selection Guide

| Model | Disk | VRAM | Speed | Accuracy | Use Case |
|-------|------|------|-------|----------|----------|
| `large-v3` | ~3GB | ~4GB | Slow | Best | Highest accuracy, batch processing |
| `medium` | ~1.5GB | ~2GB | Medium | Good | Balanced accuracy/speed |
| `small` | ~500MB | ~1GB | Fast | Fair | Real-time, lower VRAM |
| `tiny` | ~75MB | ~300MB | Fastest | Basic | Very limited VRAM, quick transcription |

> [!TIP]
> Use `--model small --model large-v3` to run both models. Clients can request either via the `model` parameter.

## Troubleshooting

| Issue | Solution |
|-------|----------|
| CUDA out of memory | Use `--device cpu` or smaller model (e.g., `--model small`) |
| Port already in use | Use `--port XXXX` to specify different port |
| Model download fails | Check network connection, or use `--download-only` first |
| Slow first request | Model is still downloading/loading. Use `--preload` to wait at startup |
| Wyoming not working | Ensure port 10300 is not blocked; check with `nc -zv localhost 10300` |

## Using with agent-cli Commands

The Whisper server is designed to work seamlessly with other agent-cli commands. See [Configuration: Local Whisper Server](../../configuration.md#using-local-whisper-server) for setup instructions.

## Installation

Requires server deps and a backend:

```bash
# faster-whisper backend (default on CUDA/CPU)
pip install "agent-cli[faster-whisper]"
# or
uv sync --extra faster-whisper
```

### macOS Apple Silicon

For optimal performance on M1/M2/M3/M4 Macs, install mlx-whisper:

```bash
pip install "agent-cli[mlx-whisper]"
```

The server will automatically detect and use the MLX backend when available.

### HuggingFace Transformers

For loading models in safetensors format (instead of CTranslate2 `.bin` files):

```bash
pip install "agent-cli[whisper-transformers]"
agent-cli server whisper --backend transformers
```

This uses HuggingFace's `transformers` library, which supports loading `.safetensors` models directly from the Hub.

For Cohere Transcribe specifically:

```bash
pip install "agent-cli[whisper-transformers]" librosa soundfile sentencepiece protobuf
agent-cli server whisper \
  --backend transformers \
  --model CohereLabs/cohere-transcribe-03-2026 \
  --default-language en
```

Notes for Cohere Transcribe:

- The HuggingFace repo is gated; you must accept the model terms on Hugging Face first.
- Set `--default-language` unless every client request will pass `language`.
- `translate`, `srt`, and `vtt` are not available for this model because it does not return timestamped segments or translation output.
- Use `--trust-remote-code` only for additional custom Hugging Face models that require repository Python code and are not recognized automatically.

### Docker

Pre-built images are available from GitHub Container Registry:

```bash
# Run with GPU support
docker run -p 10300:10300 -p 10301:10301 --gpus all ghcr.io/basnijholt/agent-cli-whisper:latest-cuda

# Run CPU-only
docker run -p 10300:10300 -p 10301:10301 ghcr.io/basnijholt/agent-cli-whisper:latest-cpu
```

Or build from source using the [`whisper.Dockerfile`](https://github.com/basnijholt/agent-cli/blob/main/docker/whisper.Dockerfile):

```bash
# Build and run with GPU support
docker build -f docker/whisper.Dockerfile --target cuda -t agent-cli-whisper:cuda .
docker run -p 10300:10300 -p 10301:10301 --gpus all agent-cli-whisper:cuda

# Build and run CPU-only
docker build -f docker/whisper.Dockerfile --target cpu -t agent-cli-whisper:cpu .
docker run -p 10300:10300 -p 10301:10301 agent-cli-whisper:cpu
```

Or use [Docker Compose](https://github.com/basnijholt/agent-cli/blob/main/docker/docker-compose.whisper.yml):

```bash
# With GPU
docker compose -f docker/docker-compose.whisper.yml --profile cuda up

# CPU only
docker compose -f docker/docker-compose.whisper.yml --profile cpu up
```

Configure via environment variables:

| Variable | Default | Description |
|----------|---------|-------------|
| `WHISPER_MODEL` | `large-v3` | Model to load |
| `WHISPER_TTL` | `300` | Seconds before unloading idle model |
| `WHISPER_DEVICE` | `cuda`/`cpu` | Device (set by target) |
| `WHISPER_LOG_LEVEL` | `info` | Logging level |
| `WHISPER_EXTRA_ARGS` | - | Additional CLI arguments |
