# Connector Progressive Rollouts

This document describes the process for managing connector progressive rollouts in Airbyte Cloud. Progressive rollouts allow gradual migration of customers to new connector versions, enabling monitoring and rollback capabilities before full release.

## Overview

Progressive rollouts provide a controlled way to release new connector versions:

- **Release Candidates (RCs)**: Pre-release versions (e.g., `0.1.0-rc.1`) that can be tested with a subset of customers
- **Gradual Migration**: Customers are progressively migrated from the stable version to the RC
- **Monitoring**: Track success/failure rates during the rollout
- **Rollback Capability**: If issues are detected, the rollout can be paused or rolled back

If a rollout exposes a bad connector version that must not be published again, use a [release block](release-blocks.md) after yanking the version.

## Progressive Rollout Lifecycle

1. [Initialize](#1-initialize) - Prepare metadata.yaml and verify rollout status
2. [Start Workflow](#2-start-workflow) - Begin the rollout workflow
3. [Manually Pin Actors](#3-manually-pin-actors-optional) - Pin specific actors for early testing (optional)
4. [Monitor](#4-monitor) - Track rollout progress and performance
5. [Pause/Resume](#5-pauseresume-optional) - Pause if issues detected (optional)
6. [Finalize](#6-finalize) - Promote, rollback, or cancel the rollout
7. [Cleanup](#7-cleanup) - Promote metadata after completion

### 1. Initialize

Prepare the connector for progressive rollout and confirm the rollout is ready to start.

**Steps:**

1. **Mark the PR for progressive rollout**: Use the `bump-version` CLI (or the `/bump-version` slash command on a PR) to bump the connector to an RC version:
   ```bash
   airbyte-ops local connector bump-version \
     --name <connector-name> \
     --repo-path /path/to/airbyte-enterprise \
     --bump-type rc  # Smart default: minor_rc if stable, bump RC number if already RC
   ```

   This command:
   - Sets `dockerImageTag` to the RC version
   - Enables `releases.rolloutConfiguration.enableProgressiveRollout: true`
   - Does not add `registryOverrides.*.dockerImageTag` pins

   The registry compiler is semver-aware, so `-rc` versions are excluded from GA/default/latest selection without override pins. The progressive rollout flag marks the version for rollout-specific handling and release-candidate metadata.

   Available bump types for RC versions: `rc` (smart default, alias for `minor_rc`), `patch_rc`, `minor_rc`, `major_rc`.

2. **Merge the PR**: Once the metadata.yaml changes are reviewed and approved, merge the PR.

3. **Verify initialization**: The connector will be automatically published after the PR is merged. Wait for the publish workflow to complete (typically a few minutes), then confirm the rollout appears in the database with "initialized" status:
   ```bash
   # Using the MCP tool
   poe mcp-tool-test query_prod_connector_rollouts '{"actor_definition_id": "<connector-uuid>"}'
   ```

   The rollout should show `state: "initialized"`.

### 2. Start Workflow

Start the rollout workflow to begin the progressive migration process.

**Tool:** `start_connector_rollout`

> [!NOTE]
> This step requires [human approval](#appendix-approval-urls).

```bash
poe mcp-tool-test start_connector_rollout '{
  "docker_repository": "airbyte/source-example",
  "docker_image_tag": "1.0.0-rc.1",
  "actor_definition_id": "<connector-uuid>",
  "approval_comment_url": "<slack-approval-url>",
  "rollout_strategy": "manual",
  "initial_rollout_pct": 25,
  "final_target_rollout_pct": 50
}'
```

**Parameters:**

- `docker_repository`: The docker repository (e.g., `airbyte/source-pokeapi`)
- `docker_image_tag`: The RC version tag (e.g., `0.3.48-rc.1`)
- `actor_definition_id`: The connector's UUID
- `approval_comment_url`: Slack approval record URL from `escalate_to_human`
- `rollout_strategy`: One of `manual`, `automated`, or `overridden` (default: `manual`)
- `customer_tier`: Optional [tier filter](./customer-tiers.md) to scope the rollout to a specific customer tier (e.g., `"TIER_0"`, `"TIER_1"`)
- `initial_rollout_pct`: Step size percentage for rollout progression (default: 25%)
- `final_target_rollout_pct`: Maximum percentage of actors to pin (default: 50%)

**Important: Multiple Calls Supported for the Same RC**

This tool can be called multiple times while the rollout is in `initialized` state to update the configuration (strategy, percentages). Once the Temporal workflow starts and the state transitions to `workflow_started`, the configuration is locked and cannot be changed.

Only one active RC is advertised per connector in compiled registry indexes, and the platform only keeps one non-terminal rollout per connector. After the PR for a newer RC is merged, the newer RC can be published and registered as a new rollout. During registration, the older rollout is marked `canceled` without unpinning. After that, when the newer rollout is started through the normal tool flow, existing rollout pins are moved to the newer RC.

<!-- TODO: Determine and document the recommended approach for configuring rollouts
https://github.com/airbytehq/airbyte-ops-mcp/issues/322 -->

**Authorization:** The start tool requires:

- `AIRBYTE_INTERNAL_ADMIN_FLAG=airbyte.io` environment variable
- A Slack approval record URL from `escalate_to_human`

After starting, the rollout state transitions to `"workflow_started"` and then `"in_progress"`.

### 3. Manually Pin Actors (Optional)

Before or during the rollout, you can manually pin specific actors (sources/destinations) to the RC version for early testing.

**Tools:** Use existing ops-mcp actor pinning tools:

- `set_workspace_connector_version_override` - Pin all actors in a workspace
- `set_cloud_connector_version_override` - Set version overrides at the cloud level

> [!NOTE]
> These tools require [human approval](#appendix-approval-urls).

This is useful for:

- Testing with internal workspaces before broader rollout
- Targeting specific customers who have agreed to beta test
- Validating fixes for customer-reported issues

### 4. Monitor

Track the rollout progress and performance of the RC version. This step is performed repeatedly throughout the rollout.

**Tool:** `query_prod_rollout_monitoring_stats`

```bash
poe mcp-tool-test query_prod_rollout_monitoring_stats '{"rollout_id": "<rollout-uuid>", "days_back": 7}'
```

**Returns:**

- **Aggregate stats:**
  - `num_actors`: Total actors using this connector
  - `num_actors_pinned`: Actors with any version pin (eligible or already pinned)
  - `num_pinned_to_rollout`: Actors specifically pinned to this rollout's RC version

- **Per-actor stats:**
  - `actor_id`: The actor UUID
  - `num_succeeded`: Number of successful syncs in the lookback period
  - `num_failed`: Number of failed syncs in the lookback period
  - `num_connections`: Number of connections using this actor

**What to look for:**

- Compare success/failure rates between RC and stable versions
- Watch for any actors with unusually high failure rates
- Monitor the progression of `num_pinned_to_rollout` as the rollout advances

### 5. Pause/Resume (Optional)

If issues are detected during monitoring, the rollout can be paused to prevent further customer migration.

**Tools:** <!-- TODO: Implement pause_rollout and resume_rollout MCP tools - https://github.com/airbytehq/airbyte-ops-mcp/issues/323 -->

```bash
# TODO: Document pause/resume commands once implemented
# https://github.com/airbytehq/airbyte-ops-mcp/issues/323
```

**When to pause:**

- Significant increase in failure rates compared to stable version
- Customer-reported issues specific to the RC version
- Infrastructure issues affecting the rollout

### 6. Finalize

Complete the rollout with one of three outcomes:

**Tool:** `finalize_connector_rollout`

> [!NOTE]
> This step requires [human approval](#appendix-approval-urls).

#### Promote to GA (succeeded)

The RC is performing well and should become the new stable version:

```bash
poe mcp-tool-test finalize_connector_rollout '{
  "docker_repository": "airbyte/source-example",
  "docker_image_tag": "1.0.0-rc.1",
  "actor_definition_id": "<connector-uuid>",
  "rollout_id": "<rollout-uuid>",
  "state": "succeeded",
  "approval_comment_url": "<slack-approval-url>"
}'
```

Do not merge an independent GA PR over an active RC rollout. If the RC should become GA, finalize the rollout as `succeeded`; the platform dispatches `finalize_rollout.yml`, which runs the promote cleanup, creates and force-merges the promotion PR, waits for the GA version to become the platform default, and then unpins actors.

#### Rollback (failed_rolled_back)

Issues were found and the RC should not be released:

```bash
poe mcp-tool-test finalize_connector_rollout '{
  "docker_repository": "airbyte/source-example",
  "docker_image_tag": "1.0.0-rc.1",
  "actor_definition_id": "<connector-uuid>",
  "rollout_id": "<rollout-uuid>",
  "state": "failed_rolled_back",
  "approval_comment_url": "<slack-approval-url>",
  "error_msg": "High failure rate detected",
  "failed_reason": "FOUND_ISSUES"
}'
```

After a rollback, yank the bad connector version if it was published. If the connector source tree still contains the bad code, add a [release block](release-blocks.md) so unrelated PRs cannot republish it before the fix lands.

#### Cancel

The rollout should be stopped without completing:

```bash
poe mcp-tool-test finalize_connector_rollout '{
  "docker_repository": "airbyte/source-example",
  "docker_image_tag": "1.0.0-rc.1",
  "actor_definition_id": "<connector-uuid>",
  "rollout_id": "<rollout-uuid>",
  "state": "canceled",
  "approval_comment_url": "<slack-approval-url>",
  "retain_pins_on_cancellation": false
}'
```

**Authorization:** The finalize tool requires:

- `AIRBYTE_INTERNAL_ADMIN_FLAG=airbyte.io` environment variable
- A Slack approval record URL from `escalate_to_human`

### 7. Cleanup

After a successful rollout, promote metadata.yaml to the GA version.

**Tool:** `airbyte-ops local connector bump-version --bump-type promote`

```bash
airbyte-ops local connector bump-version \
  --name source-github \
  --repo-path /path/to/airbyte \
  --bump-type promote
```

This step:

- Updates `dockerImageTag` to the GA version (removing `-rc.X` suffix)
- Removes or sets `releases.rolloutConfiguration.enableProgressiveRollout: false`

## Rollout States

| State | Description |
|-------|-------------|
| `initialized` | Rollout created, waiting to start |
| `workflow_started` | Rollout workflow has been triggered |
| `in_progress` | Actively migrating customers to RC |
| `paused` | Rollout temporarily stopped |
| `finalizing` | Completing the rollout |
| `succeeded` | RC promoted to GA |
| `failed_rolled_back` | RC rolled back due to issues |
| `canceled` | Rollout canceled |
| `errored` | Rollout encountered an error |

## Automated Decision Logic

The progressive rollout backend includes automated failure detection and decision-making. This section describes how the system monitors rollouts and what actions it can take.

### Signal: Sync Success/Failure Counts

The `RolloutProgressionDecider` monitors sync results from pinned connections. For each actor (source/destination) pinned to the RC version, it tracks:

- Number of successful syncs
- Number of failed syncs

### Decision States

The system can make one of four decisions:

| Decision | Description | Trigger |
|----------|-------------|---------|
| `RELEASE` | Promote RC to GA | 100% sync success rate with sufficient data |
| `ROLLBACK` | Roll back to previous version | Reserved for future use (currently triggers PAUSE) |
| `PAUSE` | Pause rollout for manual intervention | Any sync failures detected |
| `INSUFFICIENT_DATA` | Continue waiting | Not enough actors pinned or syncs completed |

### Decision Thresholds

The system uses these thresholds to make decisions:

- **Success Threshold**: 100% - All syncs must succeed for automatic promotion
- **Completion Threshold**: 50% - At least 50% of pinned actors must have completed syncs before making a final decision
- **Failure Threshold**: Any (>0) - A single failed sync triggers PAUSE

### Decision Flow

1. **Check for failures**: If any syncs have failed (`nFailedSyncs > 0`), immediately return `PAUSE`
2. **Check actor coverage**: If fewer than the required percentage of actors have completed syncs, return `INSUFFICIENT_DATA`
3. **Check success rate**: If 100% of completed syncs succeeded, return `RELEASE`
4. **Fallback**: If none of the above, return `PAUSE` for manual review

### Automatic Actions

When the workflow receives a decision:

- **RELEASE**: Finalizes the rollout with `succeeded` state, triggers GA promotion workflow
- **ROLLBACK**: Finalizes with `failed_rolled_back` state, removes RC pins (path exists but not currently auto-triggered)
- **PAUSE**: Pauses the workflow, requiring manual intervention via the `resume_rollout` or `finalize_connector_rollout` tools
- **INSUFFICIENT_DATA**: Continues the rollout loop, pinning more actors and waiting for sync results

### Rollout Expiration

If a rollout runs for too long without reaching a terminal state, it automatically pauses. This prevents rollouts from running indefinitely and ensures human review of stalled rollouts.

### Current Limitations

The system is intentionally conservative:

- **Any failure triggers PAUSE**: Rather than automatically rolling back, the system pauses to allow human judgment. This prevents premature rollbacks due to transient issues or unrelated failures.
- **100% success required**: The high threshold ensures only well-tested versions are promoted automatically.
- **Manual intervention for edge cases**: Complex scenarios (partial failures, intermittent issues) require human decision-making.

## metadata.yaml Configuration

### Progressive Rollout Fields

```yaml
data:
  # The version to be released (RC version during rollout)
  dockerImageTag: "1.0.0-rc.1"

  # Enable progressive rollout
  releases:
    rolloutConfiguration:
      enableProgressiveRollout: true
```

The registry compiler keeps active RC versions out of `latest/` default selection and exposes them separately through release-candidate metadata, so RC metadata does not need `registryOverrides.*.dockerImageTag` pins to keep the previous GA version as the default.

### Version Naming Convention

- **Stable versions**: `X.Y.Z` (e.g., `1.0.0`, `0.9.5`)
- **Release candidates**: `X.Y.Z-rc.N` (e.g., `1.0.0-rc.1`, `1.0.0-rc.2`)

When creating an RC:
- If the current version is `0.9.5`, the default RC version would be `0.10.0-rc.1`
- Multiple RCs can be created: `1.0.0-rc.1`, `1.0.0-rc.2`, etc.
- Only one RC can be active for a connector at a time. Registry compile derives it from the highest non-yanked RC version with an active `progressive-rollout.yml` marker and whose semver is greater than the latest eligible GA. Publishing a newer qualifying RC changes the advertised release candidate in the compiled registry indexes.

## Available Tools

### MCP Tools (airbyte-ops-mcp)

| Tool | Purpose |
|------|---------|
| `query_prod_connector_rollouts` | Query rollout status and history |
| `query_prod_rollout_monitoring_stats` | Get rollout progress and sync stats |
| `finalize_connector_rollout` | Promote, rollback, or cancel a rollout |
| `set_workspace_connector_version_override` | Pin actors to specific versions ([tier-gated](./customer-tiers.md)) |
| `start_connector_rollout` | Start the rollout workflow |
| `lookup_customer_tiers` | Look up [customer tier](./customer-tiers.md) for orgs, workspaces, or connections |
| `pause_rollout` | Pause an in-progress rollout <!-- TODO: Implement pause_rollout tool - https://github.com/airbytehq/airbyte-ops-mcp/issues/323 --> |
| `resume_rollout` | Resume a paused rollout <!-- TODO: Implement resume_rollout tool - https://github.com/airbytehq/airbyte-ops-mcp/issues/323 --> |
| `set_devin_reminder` | Schedule a follow-up check after a delay (e.g., health gate monitoring) |
| `escalate_to_human` | Notify a team member via Slack when human judgment is needed |

### CLI Tools (airbyte-ops)

| Command | Purpose |
|---------|---------|
| `airbyte-ops local connector bump-version --bump-type rc` | Bump to RC version and configure metadata.yaml for rollout |
| `airbyte-ops local connector bump-version --bump-type promote` | Promote RC to GA and remove rollout config |

## Agent Coordination Tools

Progressive rollouts span hours or days. Two MCP tools help AI agents manage these long-running workflows:

- **`set_devin_reminder`** — Sleep for N minutes (multiples of 30, up to 3 days), then wake to perform a follow-up action. Use this after each `progress_connector_rollout` call to schedule health gate checks once enough syncs have completed.
- **`escalate_to_human`** — Escalate to a team member via Slack (posts to `#human-in-the-loop`). Use this when a health gate fails, systematic errors appear, or human judgment is needed to decide whether to pause, rollback, or continue.

## Platform Finalization Architecture

When a rollout is finalized via `finalize_connector_rollout`, the platform takes different code paths depending on the final state. Understanding these paths is important for debugging and manual intervention.

### Finalization Code Paths

**`succeeded` or `failed_rolled_back`** → Triggers the [`finalize_rollout.yml`](https://github.com/airbytehq/airbyte/blob/master/.github/workflows/finalize_rollout.yml) GitHub Actions workflow:

1. MCP tool calls the platform's [`/connector_rollout/manual_finalize`](https://github.com/airbytehq/airbyte-platform-internal/blob/master/oss/airbyte-commons-server/src/main/kotlin/io/airbyte/commons/server/handlers/ConnectorRolloutHandlerManual.kt#L288) API endpoint
2. Handler calls [`ConnectorRolloutClient.finalizeRollout()`](https://github.com/airbytehq/airbyte-platform-internal/blob/master/oss/airbyte-connector-rollout-client/src/main/kotlin/io/airbyte/connector/rollout/client/ConnectorRolloutClient.kt#L306) which sends a Temporal workflow update
3. The Temporal workflow's [`finalizeRollout()`](https://github.com/airbytehq/airbyte-platform-internal/blob/master/oss/airbyte-connector-rollout-worker/src/main/kotlin/io/airbyte/connector/rollout/worker/ConnectorRolloutWorkflowImpl.kt#L483) method calls [`promoteOrRollbackActivity`](https://github.com/airbytehq/airbyte-platform-internal/blob/master/oss/airbyte-connector-rollout-worker/src/main/kotlin/io/airbyte/connector/rollout/worker/activities/PromoteOrRollbackActivityImpl.kt#L41)
4. `PromoteOrRollbackActivityImpl` sends an HTTP POST to the GitHub Actions [`workflow_dispatch` API](https://github.com/airbytehq/airbyte-platform-internal/blob/master/oss/airbyte-connector-rollout-worker/src/main/kotlin/io/airbyte/connector/rollout/worker/activities/PromoteOrRollbackActivityImpl.kt#L50-L108)
5. The dispatch URL is [hardcoded](https://github.com/airbytehq/airbyte-platform-internal/blob/master/oss/airbyte-connector-rollout-worker/src/main/kotlin/io/airbyte/connector/rollout/worker/runtime/AirbyteConnectorRolloutConfig.kt#L9-L10) to: `https://api.github.com/repos/airbytehq/airbyte/actions/workflows/finalize_rollout.yml/dispatches`
6. For `succeeded`: after the GHA runs, [`verifyDefaultVersionActivity`](https://github.com/airbytehq/airbyte-platform-internal/blob/master/oss/airbyte-connector-rollout-worker/src/main/kotlin/io/airbyte/connector/rollout/worker/ConnectorRolloutWorkflowImpl.kt#L512-L530) polls for up to 3 hours for the GA version to become the default, verifies that the new default matches the RC stem, then `finalizeRolloutActivity` unpins actors

### Registry Default and Superseded Detection

The platform does not proactively close an active rollout when a GA version is published independently. Normal registry publication updates the connector registry and platform default version, but it does not signal the rollout workflow or cancel active RC pins.

The legacy RC process used `registryOverrides.cloud.dockerImageTag` and `registryOverrides.oss.dockerImageTag` pins to keep the pre-RC stable version as default/latest. The old registry generation path was not semver-aware, so those overrides were necessary to keep the RC from becoming default/latest. The marker-based `airbyte-ops registry store compile` path is semver-aware: it picks the highest GA semver that has neither `version-yank.yml` nor `progressive-rollout.yml`, so `-rc` versions and rollout-marked GA versions are not eligible for default/latest selection.

RC rollout metadata no longer needs generated override pins. Versioned RC artifacts keep their RC tag, and the compiled registry exposes active rollout candidates through release-candidate metadata while default/latest stays on the previous eligible GA version. Explicit pre-existing `registryOverrides` remain manual configuration and are preserved by version-bump cleanup. Promotion finalization disables rollout metadata and renames the rollout marker so the promoted GA version becomes eligible as default/latest.


When a rollout is promoted through its own finalization path, `finalize_rollout.yml` removes the rollout metadata, promotes the RC stem to GA, force-merges the promotion PR, and publishes the registry. Do not merge an independent GA PR while the RC rollout is active; first finalize the active rollout as successful or cancel it. After promotion, `verifyDefaultVersionActivity` strips the `-rc.N` suffix from the rollout tag and compares the platform default against that stem:

- `1.2.3-rc.1` promoted and platform default becomes `1.2.3`: rollout remains `succeeded`
- `1.2.3-rc.1` promoted but platform default becomes `1.2.4`: rollout is marked `canceled` with "Default version is not the release candidate; rollout was superseded"
- `1.2.3-rc.1` is still running and an unrelated PR publishes `1.2.4`: rollout state does not change immediately; it can only be marked superseded later if it finalizes as promoted and observes `1.2.4` as the default
- `1.2.3-rc.1` is still running and an independent PR publishes matching GA `1.2.3`: rollout state does not change immediately; if it later finalizes as promoted and observes `1.2.3` as the default, the stem check passes

Actors pinned by the rollout remain pinned to the RC until the rollout reaches a terminal path that unpins them or until a manual cancel path removes pins.

### Successive RCs for the Same Connector

After the PR for a newer RC is merged, the newer RC can be published, become the active release candidate in compiled registry indexes, and create a new rollout record for the newer RC. During rollout insertion, the platform cancels any existing non-terminal rollout for the same actor definition with `Rollout was incomplete when a new rollout was started. Cancelled without unpinning.`

The older rollout's actor pins are intentionally retained at cancellation time. The normal `start_connector_rollout` tool flow uses the platform start API's default `migrate_pins=true` behavior; the start path rewrites rollout-origin pins from previous rollout IDs to the newer rollout ID and updates their version value to the newer RC version. If a custom caller explicitly disables `migrate_pins`, actors pinned by the older rollout remain pinned to the older RC until explicitly unpinned or migrated.

If the same RC version is seen again and an existing non-canceled rollout already exists for that actor definition and release candidate version ID, the platform does not create a duplicate rollout.

**`canceled`** → Does **NOT** trigger the GitHub workflow:

1. Handler calls [`cancelRollout()`](https://github.com/airbytehq/airbyte-platform-internal/blob/master/oss/airbyte-commons-server/src/main/kotlin/io/airbyte/commons/server/handlers/ConnectorRolloutHandlerManual.kt#L326) directly
2. Sets DB state to `canceled` and removes RC actor pins
3. Calls [`ConnectorRolloutClient.cancelRollout()`](https://github.com/airbytehq/airbyte-platform-internal/blob/master/oss/airbyte-connector-rollout-client/src/main/kotlin/io/airbyte/connector/rollout/client/ConnectorRolloutClient.kt#L359) which **terminates** the Temporal workflow
4. No GitHub workflow runs; the next registry compile determines whether the RC is still advertised from versioned marker files

### Registry marker files

Progressive rollout registry state is expressed through exact marker filenames in the versioned GCS directory:

- `progressive-rollout.yml` -- active rollout marker. Publish creates this when generated metadata has `enableProgressiveRollout: true`.
- `progressive-rollout-promoted-yyyymmdd.yml` -- inactive audit marker after promotion.
- `progressive-rollout-aborted-yyyymmdd.yml` -- inactive audit marker after rollback/abort.
- `version-yank.yml` -- absolute withdrawal marker. If present, compile excludes the version before evaluating progressive rollout state.

Compile only treats exact active filenames as state. Dated marker files are audit history and are ignored by latest/RC selection.

### GitHub Workflow Dispatch Details

The platform triggers `finalize_rollout.yml` via the GitHub Actions `workflow_dispatch` API. The workflow also declares `repository_dispatch` for compatibility, but `PromoteOrRollbackActivityImpl` sends:

```json
{
  "ref": "master",
  "inputs": {
    "connector_name": "<technical-name>",
    "action": "<promote-or-rollback>"
  }
}
```

- **Cadence:** On-demand, triggered by the Temporal `PromoteOrRollbackActivityImpl` activity
- **Only for:** `succeeded` (action=`promote`) and `failed_rolled_back` (action=`rollback`)
- **Not for:** `canceled` — cancellation skips the GHA entirely

<details>
<summary>Legacy `release_candidate/` marker cleanup</summary>

Older registry workflows used a stateful `release_candidate/` GCS directory to advertise the active RC. `publish_connectors.yml` created it with `airbyte-ops registry progressive-rollout create`, and `finalize_rollout.yml` removed it with `airbyte-ops registry progressive-rollout cleanup`.

Those commands and workflow steps have been removed. `registry compile` now derives active RCs from version-level marker files. A legacy `release_candidate/` directory may still appear in old buckets or logs, but it is no longer an input to compiled registry indexes.

</details>

## Related Resources

- [Airbyte Ops MCP Repository](https://github.com/airbytehq/airbyte-ops-mcp)
- [Connector Registry](https://connectors.airbyte.com/files/registries/v0/oss_registry.json)
- [`finalize_rollout.yml` workflow](https://github.com/airbytehq/airbyte/blob/master/.github/workflows/finalize_rollout.yml)
- [`publish_connectors.yml` workflow](https://github.com/airbytehq/airbyte/blob/master/.github/workflows/publish_connectors.yml)
- [Platform rollout worker code](https://github.com/airbytehq/airbyte-platform-internal/tree/master/oss/airbyte-connector-rollout-worker)
- [Regression Tests](./regression-tests.md) — validating connector behavior before and during rollouts
- <!-- TODO: Add link to internal rollout dashboard if available - https://github.com/airbytehq/airbyte-ops-mcp/issues/322 -->
- <!-- TODO: Add link to connector publishing documentation - https://github.com/airbytehq/airbyte-ops-mcp/issues/322 -->

## Appendix: Approval URLs

Certain high-impact operations require an approval URL as an authorization mechanism. This provides an audit trail and ensures that sensitive actions are explicitly approved by an authorized team member.

### How It Works

An approval URL is a Slack approval record URL from the `escalate_to_human` MCP tool. The system resolves the approving user through the team roster before allowing the operation to proceed.

**Example approval URL:**
```
https://airbyteio.slack.com/archives/<channel-id>/p<message-id>
```

### Creating an Approval Record

To create a valid approval record:

1. Call `escalate_to_human` with `approval_requested=True` and a summary of the planned action
2. Wait for the user to approve the Slack message
3. Pass the approval record URL as the `approval_comment_url` parameter

### Actions Requiring Approval

| Action | Tool | Why Approval Is Required |
|--------|------|--------------------------|
| Pin workspace actors | `set_workspace_connector_version_override` | Affects all actors in a workspace |
| Pin cloud actors | `set_cloud_connector_version_override` | Affects actors at the cloud level |
| Start rollout workflow | `start_connector_rollout` | Initiates customer migration to RC version |
| Promote RC to GA | `finalize_connector_rollout` (state: `succeeded`) | Affects all customers using the connector |
| Rollback RC | `finalize_connector_rollout` (state: `failed_rolled_back`) | Reverts customers to previous version |
| Cancel rollout | `finalize_connector_rollout` (state: `canceled`) | Terminates the rollout process |

### Additional Requirements

In addition to the approval URL, these operations also require the `AIRBYTE_INTERNAL_ADMIN_FLAG=airbyte.io` environment variable to be set. This two-factor authorization ensures that both the tool operator and an @airbyte.io team member have approved the action.

## Troubleshooting

### Rollout not appearing after merge

**Symptom:** The connector PR was merged but no rollout appears in the database.

**Possible causes:**
1. The connector publish workflow hasn't completed yet
2. The publish workflow did not create `progressive-rollout.yml`
3. The connector version wasn't properly tagged as an RC
4. The RC version is not newer than the latest non-yanked GA version

**Resolution:**
- Wait for the publish workflow to complete (check GitHub Actions)
- Verify metadata.yaml has `releases.rolloutConfiguration.enableProgressiveRollout: true`
- Verify the published RC directory has an active `progressive-rollout.yml` marker
- Ensure `dockerImageTag` follows the RC naming convention (`X.Y.Z-rc.N`)
- Ensure the RC semver is greater than the latest non-yanked GA semver

### High failure rate during rollout

**Symptom:** Monitoring shows significantly higher failure rates for the RC compared to stable.

**Resolution:**
1. Pause the rollout to prevent further customer impact
2. Investigate the failures using job logs
3. If issues are confirmed, finalize with `failed_rolled_back` state
4. Fix the issues and create a new RC (`X.Y.Z-rc.2`)

### Cannot finalize rollout

**Symptom:** The finalize tool returns an authorization error.

**Possible causes:**
1. Missing `AIRBYTE_INTERNAL_ADMIN_FLAG` environment variable
2. Invalid or expired Slack approval record URL
3. Approver cannot be resolved to an @airbyte.io user

**Resolution:**
- Ensure `AIRBYTE_INTERNAL_ADMIN_FLAG=airbyte.io` is set
- Get a fresh Slack approval record from an @airbyte.io team member
- Verify the approval URL is accessible and the approver can be resolved

### Actors not being migrated

**Symptom:** `num_pinned_to_rollout` stays at 0 even after starting the workflow.

**Possible causes:**
1. Rollout is still in `initialized` state (workflow not started)
2. Rollout percentage is set to 0
3. No eligible actors for this connector

**Resolution:**
- Verify the rollout state is `in_progress`
- Check the `current_target_rollout_pct` value
- Confirm there are active actors using this connector (`num_actors` > 0)
