# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

AgileRL is a deep reinforcement learning library focused on RLOps - MLOps for reinforcement learning. The key innovation is evolutionary hyperparameter optimization (HPO) that automatically tunes hyperparameters during training, reducing the need for separate HPO experiments.

## Development Commands

```bash
# Install in development mode
pip install -e .

# Install with LLM dependencies (for GRPO, DPO algorithms)
pip install -e ".[all]"

# Run all tests
pytest tests

# Run tests with coverage
pytest --cov=agilerl --cov-report=xml

# Run a single test file
pytest tests/test_algorithms/test_dqn.py

# Run a specific test
pytest tests/test_algorithms/test_dqn.py::test_dqn_init -v

# Skip LLM-related tests (faster)
pytest -m "not llm"

# Run tests with fail-fast
pytest --exitfirst

# Lint and format (via pre-commit)
pre-commit run --all-files

# Install pre-commit hooks
pre-commit install
```

## Architecture

### Core Algorithm Hierarchy

All RL algorithms inherit from a common base in `agilerl/algorithms/core/base.py`:

- **EvolvableAlgorithm**: Base metaclass providing mutation registry, checkpointing, cloning, and evolutionary features
- **RLAlgorithm(EvolvableAlgorithm)**: Single-agent RL base with observation preprocessing and action handling
- **MultiAgentRLAlgorithm(EvolvableAlgorithm)**: Multi-agent RL base supporting agent-specific networks

### Evolvable Networks System

The evolution system uses two core concepts in `agilerl/modules/`:

- **EvolvableModule** (`modules/base.py`): Base class for neural network modules that support architectural mutations (add/remove layers, nodes, change activations). Uses `@mutation` decorator to mark methods that trigger network reconstruction.
- **EvolvableNetwork** (`networks/base.py`): Encoder-head architecture combining feature extraction with task-specific heads. Mutations can target encoder or head independently.

Network configurations are dataclasses in `agilerl/modules/configs.py` (MlpNetConfig, CnnNetConfig, etc.).

### HPO Components (`agilerl/hpo/`)

- **Mutations** (`mutation.py`): Applies architectural and hyperparameter mutations to agent populations
- **TournamentSelection** (`tournament.py`): Selects fittest agents based on evaluation scores

### Training Loop Functions (`agilerl/training/`)

Pre-built training loops that handle the full evolutionary training cycle:
- `train_off_policy`: DQN, DDPG, TD3, Rainbow
- `train_on_policy`: PPO
- `train_multi_agent_off_policy`: MADDPG, MATD3
- `train_multi_agent_on_policy`: IPPO
- `train_bandits`: NeuralUCB, NeuralTS
- `train_offline`: CQL
- `train_llm`: GRPO, DPO

### Registry System (`agilerl/algorithms/core/registry.py`)

Algorithms declare their evolvable components via `MutationRegistry`:
- **NetworkGroup**: Groups related networks (e.g., actor/critic, eval/target)
- **OptimizerConfig**: Maps optimizers to networks with hyperparameter bounds
- **HyperparameterConfig**: Defines mutable RL hyperparameters (lr, batch_size, etc.)

### Protocol Types (`agilerl/protocols.py`)

Runtime-checkable protocols define interfaces for:
- `EvolvableAlgorithmProtocol`: Required methods for evolutionary algorithms
- `EvolvableModuleProtocol`: Required methods for evolvable network modules
- `EvolvableNetworkProtocol`: Encoder-head network interface

## Key Patterns

### Creating Populations

```python
from agilerl.utils.utils import create_population

pop = create_population(
    algo="DQN",
    observation_space=obs_space,
    action_space=act_space,
    INIT_HP=init_hp_dict,
    population_size=6,
    device="cuda"
)
```

### Mutation Flow

1. Tournament selection picks winners from population
2. Clone winners to replace losers
3. Apply mutations via `Mutations.mutation()`:
   - Network architecture mutations (layers, nodes)
   - Hyperparameter mutations (lr, batch_size)
4. Rebuild optimizers for mutated networks

### Multi-Agent Support

Uses PettingZoo-style parallel API. Agent networks can be:
- **Homogeneous**: Shared architecture across agents
- **Heterogeneous**: Per-agent architectures via `ModuleDict`

## Testing Conventions

- Test files mirror source structure: `tests/test_<module>/test_<file>.py`
- Common fixtures in `tests/conftest.py` (spaces, networks, configs)
- Helper functions in `tests/helper_functions.py`
- LLM tests marked with `@pytest.mark.llm`
- Tests use session-scoped fixtures for spaces to avoid recreation overhead

## Branch Workflow

- `main`: Stable releases
- `nightly`: Active development (PRs target this branch)

## Linting

Uses Ruff for linting and formatting. Configuration in `pyproject.toml` with relaxed rules for test files.
