# Connector Regression Tests

This document describes the connector regression testing system in `airbyte-ops-mcp`. Regression tests validate connector behavior by running Airbyte protocol commands and checking the output. They support both single-version validation and comparison testing against a baseline version to detect regressions.

## Recommended Usage

The primary and recommended way to run regression tests is via the [`/ai-prove-fix` slash command](https://github.com/airbytehq/ai-skills/blob/main/devin/playbooks/prove_fix.md) on a connector PR. This triggers a Devin session that customizes the test execution based on the PR context and intelligently parses and reviews the outputs.

Under the hood, `/ai-prove-fix` uses the `run_regression_tests` MCP tool described below. The tool can also be called directly by AI agents for programmatic triggering. AI agents also have access to the [`connector-regression-tests` skill](https://github.com/airbytehq/ai-skills/blob/main/.agents/skills/connector-regression-tests/SKILL.md), which provides step-by-step guidance for running and interpreting regression tests.

The `run_regression_tests` MCP tool triggers the [GitHub Actions workflow](../.github/workflows/connector-regression-test.yml) for PR-based regression testing. The underlying `airbyte-ops cloud connector regression-test` CLI command runs the regression test logic locally; in CI, that same CLI command is invoked by the workflow after the workflow checks out and builds the connector under test.

## Test Modes

### Comparison Mode (default)

Runs the connector against both a **target** (new) and **control** (baseline) version, then compares results to detect regressions.

- **Target version**: Built from the PR branch (new code)
- **Control version**: The current released version (baseline), auto-detected from `metadata.yaml`
- A **regression** is flagged when the target fails but the control succeeds
- If both versions fail, that indicates an environment issue rather than a regression

### Single-Version Mode

Set `skip_compare=True` to run tests against a single version without comparison. Useful for quick validation or when no baseline version exists.

## Protocol Commands

The workflow runs up to four Airbyte protocol commands sequentially:

| Command | Timeout | Description |
|---------|---------|-------------|
| `SPEC` | 30 min | Validates connector spec output |
| `CHECK` | 30 min | Validates connection check with config |
| `DISCOVER` | 60 min | Validates catalog discovery |
| `READ` | 180 min | Validates data reading (skippable with `skip_read_action`) |

## MCP Tool: `run_regression_tests`

### Parameters

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `connector_name` | string | Yes | -- | Connector name (e.g., `source-pokeapi`) |
| `pr` | int | Yes | -- | PR number to checkout and build from |
| `repo` | enum | Yes | -- | `airbyte` for OSS or `airbyte-enterprise` for enterprise connectors |
| `connection_id` | string | No | `None` | Airbyte Cloud connection ID to fetch config/catalog from. If omitted, uses GSM integration test secrets |
| `skip_compare` | bool | No | `False` | `True` for single-version mode, `False` for comparison mode |
| `skip_read_action` | bool | No | `False` | `True` to skip the READ command (run only spec, check, discover) |
| `override_test_image` | string | No | `None` | Override target connector image with a tag (e.g., `airbyte/source-github:1.0.0`). Ignored if `skip_compare=False` |
| `override_control_image` | string | No | `None` | Override control (baseline) connector image with a tag. Ignored if `skip_compare=True` |
| `workspace_id` | string | No | `None` | Workspace ID (UUID) or alias (e.g., `@devin-ai-sandbox`). Validates that the connection belongs to this workspace |
| `selected_streams` | list | No | `None` | List of stream names to test. Limits the configured catalog to reduce data volume |
| `enable_debug_logs` | bool | No | `False` | Enable debug-level logging for both the test harness and the connector container (`LOG_LEVEL=DEBUG`) |

### Return Value

Returns a response with:

- `run_id` -- unique tracking identifier
- `status` -- `queued` or `failed`
- `message` -- human-readable status message
- `workflow_url` -- URL to the GitHub Actions workflow
- `github_run_id` -- workflow run ID for polling status
- `github_run_url` -- direct URL to the workflow run

### Config and Credential Sources

There are multiple ways to provide connector configuration:

1. **`connection_id`** -- Fetches config and catalog from an existing Airbyte Cloud connection. This allows testing against live customer data and is the preferred approach when validating fixes for specific connections.
2. **GSM secrets** (default) -- Fetches integration test credentials from Google Secret Manager via PyAirbyte when no `connection_id` is provided

## Validation Layers

> **Note:** The validators and comparators below are available as library functions in
> `regression_tests/validation/` and `regression_tests/regression/`. They are not
> currently invoked by the workflow/CLI path, which determines pass/fail from
> container exit codes. They are documented here as reference for future integration
> and for callers that use the library directly.

### Catalog Validations (DISCOVER output)

| Validation | Evaluation Mode | Description |
|------------|-----------------|-------------|
| Has streams | Strict | Catalog contains at least one stream |
| No duplicate stream names | Strict | All stream names are unique |
| Valid JSON Schema | Strict | All schemas are valid JSON Schema Draft 7 |
| Cursor fields exist | Diagnostic | Cursor fields exist in their stream schemas |
| No unresolved `$ref` | Strict | No unresolved JSON Schema `$ref` values |
| No disallowed keywords | Diagnostic | No `allOf` or `not` keywords in schemas |
| Primary keys exist | Strict | Primary key fields exist in schemas |
| Sync modes defined | Strict | All streams have sync modes |
| `additionalProperties` is `true` | Diagnostic | `additionalProperties` set to `true` for backward compatibility |

### Record Validations (READ output)

| Validation | Evaluation Mode | Description |
|------------|-----------------|-------------|
| Has records | Strict | At least one record was read |
| Records conform to schema | Strict | All records match their stream schemas |
| Primary keys in records | Strict | All records have non-null primary key values |
| State messages emitted | Strict | State messages emitted for each stream |

### Comparison Validations (comparison mode only)

| Comparator | Description |
|------------|-------------|
| Record counts | Target has at least as many records as control |
| Primary keys | All primary keys from control are present in target |
| Full record diff | All records are identical (excluding `emitted_at` timestamps) |
| Schema comparison | Inferred schemas match between versions |

## Evaluation Modes

- **Strict** -- Test failures cause the overall test run to fail. Used for critical validations.
- **Diagnostic** -- Failures are surfaced in the report but do not fail the overall run. Used for best-practice checks that don't affect core functionality.

## Artifacts

Each test run uploads the following artifacts to GitHub Actions (7-day retention):

- `stdout.txt` / `stderr.txt` per command execution
- Airbyte messages split by type (`.jsonl` files)
- Configured catalog JSON
- Regression report (`report.md`)
- HTTP traffic dumps (if `enable_http_metrics` is enabled via CLI)

## Related Resources

- [`/ai-prove-fix` playbook](https://github.com/airbytehq/ai-skills/blob/main/devin/playbooks/prove_fix.md) -- the recommended slash command for running regression tests
- [`connector-regression-tests` skill](https://github.com/airbytehq/ai-skills/blob/main/.agents/skills/connector-regression-tests/SKILL.md) -- step-by-step skill for AI agents running regression tests
- [Progressive Rollouts](./progressive-rollouts.md) -- managing controlled connector releases
- [Connector Registry](./connector-registry.md) -- registry operations and metadata
- [Customer Tiers](./customer-tiers.md) -- tier-based scoping for test connections

## Appendix: CLI Reference

The CLI command `airbyte-ops cloud connector regression-test` provides the same functionality as the MCP tool and is used internally by CI. It is **not recommended for direct human use** -- prefer the `/ai-prove-fix` slash command instead.

The CLI accepts additional options not exposed through the MCP tool:

- `--config-path`, `--catalog-path`, `--state-path` -- local JSON files for connector configuration
- `--enable-http-metrics` -- capture HTTP traffic dumps
- `--repo-root` -- path to the local repository root
