---
name: progressive-rollout-e2e-test
description: End-to-end test of the connector progressive rollout lifecycle — RC creation via slash command, publish, platform initialization, rollback, second RC, and promote. Use when validating that the finalize_rollout workflow correctly handles promote and rollback actions.
devin-enabled: true
icon: flask-conical
---

# Progressive Rollout E2E Test

Run a full end-to-end test of the connector progressive rollout lifecycle. This test exercises the entire pipeline from RC version creation through platform initialization, rollback, re-creation, and promotion — validating the `finalize_rollout.yml` workflow and all supporting CLI commands.

## Prerequisites

- **Repos:** `~/repos/airbyte` (connector source, slash commands) and `~/repos/airbyte-ops-mcp` (ops CLI source)
- **GitHub write access:** To `airbytehq/airbyte` (for creating PRs, running slash commands, and merging)
- **Credentials:** Airbyte Cloud API env vars for MCP tool calls:
  - `AIRBYTE_CLOUD_CLIENT_ID`
  - `AIRBYTE_CLOUD_CLIENT_SECRET`
  - `AIRBYTE_INTERNAL_ADMIN_FLAG` (must be set to `airbyte.io` for `finalize_connector_rollout`)
- **Credentials:** `GCS_CREDENTIALS` or `GCP_GSM_CREDENTIALS` for registry operations

## Devin Secrets Needed

- `GCP_GSM_CREDENTIALS` — Service account JSON key with GCS bucket access
- Airbyte Cloud API env vars for MCP tools (especially `finalize_connector_rollout`):
  - `AIRBYTE_CLOUD_CLIENT_ID`
  - `AIRBYTE_CLOUD_CLIENT_SECRET`
  - `AIRBYTE_INTERNAL_ADMIN_FLAG=airbyte.io`

## Test Connector

Use `source-faker` throughout this test:
- **Connector name:** `source-faker`
- **Docker repository:** `airbyte/source-faker`
- **Actor definition ID:** `dfd88b22-b603-4c3d-aad7-3701784586b1`
- **Current version (as of writing):** `7.0.2`

Record the current version from `metadata.yaml` before starting — the test plan references it as `CURRENT_VERSION`.

## Step 0: Collect Approval URL

Before any finalize calls, collect a single approval URL that will be reused for all finalize operations in this test. Use the `escalate_to_human` MCP tool to send an approval request to the user via Slack:

```bash
uv run poe mcp-tool-test escalate_to_human '{
  "target_person": "<USER_EMAIL>",
  "message": "Requesting approval for progressive rollout E2E test on source-faker. This will create RC versions, test rollback, and test promote flows.",
  "agent_session_url": "<DEVIN_SESSION_URL>",
  "approval_requested": true,
  "approval_request_summary": "Progressive rollout E2E test: create RC, rollback, create RC2, promote on source-faker",
  "connector_name": "source-faker",
  "request_type": "approval"
}'
```

Wait for the user to click "Approve" in the Slack message. Once approved, the Slack message URL becomes the `APPROVAL_COMMENT_URL`. Record it — you will pass it to both `finalize_connector_rollout` calls (Phase 2 and Phase 4).

---

## Phase 1: Create RC1

### Step 1.1: Create a PR that modifies source-faker

Create a branch off `master` and make a minor change to `source-faker` (e.g., add a comment to a file, or update the description in `metadata.yaml`). Open a PR.

```bash
cd ~/repos/airbyte
git checkout master && git pull
git checkout -b test/progressive-rollout-e2e-source-faker
# Make a trivial change to source-faker (e.g., add a comment)
echo "# Progressive rollout e2e test $(date +%Y%m%d)" >> airbyte-integrations/connectors/source-faker/README.md
git add airbyte-integrations/connectors/source-faker/README.md
git commit -m "test: progressive rollout e2e test setup"
git push origin test/progressive-rollout-e2e-source-faker
```

Create the PR via `git_create_pr`. Record the **PR number** as `TEST_PR_NUMBER`.

### Step 1.2: Run `/bump-progressive-rollout-version` slash command

Post a comment on the PR:

```
/bump-progressive-rollout-version type=minor
```

This triggers the `bump-progressive-rollout-version-command.yml` workflow, which will:
1. Run `airbyte-ci connectors --modified bump-version minor --rc` on the PR branch
2. Commit the version bump back to the PR branch

**Wait** for the workflow to complete. Check the PR comments for the success/failure message.

**Expected result:**
- `metadata.yaml` updated with:
  - `dockerImageTag: <NEXT_MINOR>-rc.1` (e.g., `7.1.0-rc.1`)
  - `registryOverrides.cloud.dockerImageTag: <CURRENT_VERSION>` (e.g., `7.0.2`)
  - `registryOverrides.oss.dockerImageTag: <CURRENT_VERSION>` (e.g., `7.0.2`)
  - `releases.rolloutConfiguration.enableProgressiveRollout: true`
- Changelog entry added for the new version

Record the **RC version** as `RC1_VERSION` (e.g., `7.1.0-rc.1`).

### Step 1.3: Merge the PR

Merge the PR to `master`. This triggers the publish workflow.

```bash
# Use gh CLI or merge via GitHub
gh pr merge TEST_PR_NUMBER --squash
```

### Step 1.4: Wait for publish workflow to complete

The publish workflow (`publish_connectors.yml`) will:
1. Build and publish the connector Docker image with the RC tag
2. Generate and publish registry artifacts
3. Create the `release_candidate/` metadata marker in GCS (because the version contains `-rc.`)
4. Recompile the registry

**Wait** for the publish workflow to complete (check GitHub Actions). This typically takes 5-15 minutes.

### Step 1.5: Verify RC1 is initialized in prod database

Query the prod database for the rollout status:

```bash
cd ~/repos/airbyte-ops-mcp
uv run poe mcp-tool-test query_prod_connector_rollouts '{
  "actor_definition_id": "dfd88b22-b603-4c3d-aad7-3701784586b1",
  "active_only": true
}'
```

**Expected result:**
- A rollout record with `state: "initialized"`
- `rc_docker_image_tag` matching `RC1_VERSION`
- `rc_docker_repository: "airbyte/source-faker"`

Record the **rollout ID** as `ROLLOUT1_ID`.

**If no rollout appears after 15 minutes, check:**
- Did the publish workflow succeed?
- Was the RC metadata marker created in GCS?
- Does the version in metadata.yaml contain `-rc.`?

---

## Phase 2: Rollback RC1

### Step 2.1: Cancel the rollout via API

Use the `finalize_connector_rollout` MCP tool to cancel the rollout (which triggers rollback behavior):

```bash
uv run poe mcp-tool-test finalize_connector_rollout '{
  "docker_repository": "airbyte/source-faker",
  "docker_image_tag": "<RC1_VERSION>",
  "actor_definition_id": "dfd88b22-b603-4c3d-aad7-3701784586b1",
  "rollout_id": "<ROLLOUT1_ID>",
  "state": "canceled",
  "approval_comment_url": "<APPROVAL_COMMENT_URL>"
}'
```

Use the `APPROVAL_COMMENT_URL` collected in Step 0.

**Important:** The `finalize_connector_rollout` API call triggers the platform's Temporal workflow (`PromoteOrRollbackActivityImpl`), which dispatches the `finalize_rollout.yml` GitHub Actions workflow with `action=rollback`.

### Step 2.2: Verify rollback workflow runs

Check GitHub Actions for a `Finalize Progressive Rollout` workflow run with `action=rollback`:

```
https://github.com/airbytehq/airbyte/actions/workflows/finalize_rollout.yml
```

**Expected behavior (Job 1: `rc-registry-cleanup`):**
1. `airbyte-ops registry rc cleanup` deletes the `release_candidate/` marker from GCS
2. `airbyte-ops registry store compile --connector-name source-faker` recompiles the registry
3. Slack notification sent to `#connector-publish-failures` with ↩️ emoji

**Expected behavior (Job 2: `create-promotion-pr`):**
- **SKIPPED** — Job 2 has an `if` condition that only runs for `action == 'promote'`

### Step 2.3: Verify rollback results

After the workflow completes:

1. **GCS:** The `release_candidate/` marker should be deleted:
   ```bash
   # This should return NOT FOUND
   gsutil ls "gs://prod-airbyte-cloud-connector-metadata-service/metadata/airbyte/source-faker/release_candidate/" 2>&1
   ```

2. **Git:** No changes to `master` — the RC version (`RC1_VERSION`) and `enableProgressiveRollout: true` remain in `metadata.yaml`. This is by design.

3. **Prod DB:** The rollout should be in a terminal state (`canceled` or `failed_rolled_back`):
   ```bash
   uv run poe mcp-tool-test query_prod_connector_rollouts '{
     "rollout_id": "<ROLLOUT1_ID>"
   }'
   ```

4. **Slack:** Check `#connector-publish-failures` for the rollback success message.

---

## Phase 3: Create RC2

### Step 3.1: Create a new PR for RC2

Create a new branch and PR:

```bash
cd ~/repos/airbyte
git checkout master && git pull
git checkout -b test/progressive-rollout-e2e-source-faker-rc2
# Make another trivial change
echo "# RC2 progressive rollout e2e test $(date +%Y%m%d)" >> airbyte-integrations/connectors/source-faker/README.md
git add airbyte-integrations/connectors/source-faker/README.md
git commit -m "test: progressive rollout e2e test RC2"
git push origin test/progressive-rollout-e2e-source-faker-rc2
```

Create the PR. Record the **PR number** as `TEST_PR2_NUMBER`.

### Step 3.2: Run `/bump-progressive-rollout-version` again

Post a comment on the new PR:

```
/bump-progressive-rollout-version type=minor
```

**Expected result:**
- Since the previous RC version (`RC1_VERSION`, e.g., `7.1.0-rc.1`) is still in `metadata.yaml` on `master`, the next bump should produce `RC1_BASE-rc.2` (e.g., `7.1.0-rc.2`).
- `registryOverrides` should still pin to the original stable version (`CURRENT_VERSION`).

Record the **RC2 version** as `RC2_VERSION` (e.g., `7.1.0-rc.2`).

### Step 3.3: Merge the PR

Merge the PR to `master`.

### Step 3.4: Wait for publish and verify initialization

Same as Phase 1 Steps 1.4 and 1.5:
1. Wait for publish workflow to complete
2. Query prod DB for the new rollout

```bash
uv run poe mcp-tool-test query_prod_connector_rollouts '{
  "actor_definition_id": "dfd88b22-b603-4c3d-aad7-3701784586b1",
  "active_only": true
}'
```

**Expected:** New rollout with `state: "initialized"` and `rc_docker_image_tag: RC2_VERSION`.

Record the **rollout ID** as `ROLLOUT2_ID`.

---

## Phase 4: Promote RC2

### Step 4.1: Promote the rollout via API

Use the `finalize_connector_rollout` MCP tool to promote:

```bash
uv run poe mcp-tool-test finalize_connector_rollout '{
  "docker_repository": "airbyte/source-faker",
  "docker_image_tag": "<RC2_VERSION>",
  "actor_definition_id": "dfd88b22-b603-4c3d-aad7-3701784586b1",
  "rollout_id": "<ROLLOUT2_ID>",
  "state": "succeeded",
  "approval_comment_url": "<APPROVAL_COMMENT_URL>"
}'
```

Use the same `APPROVAL_COMMENT_URL` from Step 0.

This triggers the `finalize_rollout.yml` workflow with `action=promote`.

### Step 4.2: Verify promote workflow runs

Check GitHub Actions for a `Finalize Progressive Rollout` workflow run with `action=promote`.

**Expected behavior (Job 1: `rc-registry-cleanup`):**
1. `airbyte-ops registry rc cleanup` deletes the `release_candidate/` marker
2. `airbyte-ops registry store compile --connector-name source-faker` recompiles
3. No Slack rollback message (rollback notification only fires for `action == rollback`)

**Expected behavior (Job 2: `create-promotion-pr`):**
1. Checks out the repo as OCTAVIA_BOT
2. Runs `airbyte-ops local connector bump-version --repo-path . --name source-faker --bump-type promote --no-changelog`
   - Strips `-rc.2` suffix → GA version (e.g., `7.1.0`)
   - Sets `enableProgressiveRollout: false`
   - Removes `registryOverrides.cloud.dockerImageTag` and `registryOverrides.oss.dockerImageTag`
3. First `create-pull-request` call creates a draft PR with version bump changes
4. `airbyte-ops local connector changelog add` writes changelog entry with PR number
5. Second `create-pull-request` call pushes changelog commit to same PR
6. `gh pr ready` + `gh pr merge --squash --admin` force-merges the PR
7. Slack notification with ↗️ emoji

### Step 4.3: Verify promote results

After the workflow completes:

1. **PR:** A merged PR should exist titled `chore: finalize promote for source-faker`:
   ```bash
   gh pr list --repo airbytehq/airbyte --state merged --search "finalize promote source-faker" --limit 5
   ```

2. **metadata.yaml on master:** Pull latest and verify:
   ```bash
   cd ~/repos/airbyte && git checkout master && git pull
   cat airbyte-integrations/connectors/source-faker/metadata.yaml
   ```
   - `dockerImageTag` should be the GA version (e.g., `7.1.0`) — no `-rc.N` suffix
   - `enableProgressiveRollout` should be `false`
   - `registryOverrides.cloud.dockerImageTag` and `registryOverrides.oss.dockerImageTag` should be removed (or absent)

3. **Changelog:** The connector's doc file should have a new changelog entry:
   ```bash
   grep "7.1.0" docs/integrations/sources/faker.md
   ```
   - Should show `Promoted release candidate to GA` with the PR number

4. **GCS:** The `release_candidate/` marker should be deleted.

5. **Prod DB:** The rollout should be in `succeeded` state:
   ```bash
   uv run poe mcp-tool-test query_prod_connector_rollouts '{
     "rollout_id": "<ROLLOUT2_ID>"
   }'
   ```

6. **Slack:** Check `#connector-publish-failures` for the promote success message (↗️).

7. **Publish pipeline:** After the force-merged PR triggers publish, the platform's `verifyDefaultVersionActivity` should detect the new GA version and complete the rollout lifecycle.

---

## Cleanup

After the test is complete:

1. **Delete test branches** (if not auto-deleted):
   ```bash
   git push origin --delete test/progressive-rollout-e2e-source-faker 2>/dev/null || true
   git push origin --delete test/progressive-rollout-e2e-source-faker-rc2 2>/dev/null || true
   ```

2. **Verify source-faker is in a clean state** — the promote should have left it with:
   - A clean GA version in `metadata.yaml`
   - `enableProgressiveRollout: false`
   - No `registryOverrides` version pins

If the test was aborted mid-way:
- If an RC is still active, cancel it via `finalize_connector_rollout` with `state: "canceled"`
- If `metadata.yaml` has stale RC config, create a manual cleanup PR

---

## Pass/Fail Criteria

### Overall PASS requires ALL of:

- [ ] Phase 1: RC1 created via `/bump-progressive-rollout-version`, published, and initialized in prod DB
- [ ] Phase 2: Rollback workflow triggered, Job 1 ran (GCS cleanup), Job 2 skipped, Slack notified
- [ ] Phase 2: No git changes on master after rollback (RC version persists — by design)
- [ ] Phase 3: RC2 created, published, and initialized in prod DB
- [ ] Phase 4: Promote workflow triggered, both jobs ran, PR created and force-merged
- [ ] Phase 4: `metadata.yaml` on master has clean GA version, no RC suffix, no progressive rollout flag
- [ ] Phase 4: Changelog entry includes correct GA version and PR number
- [ ] Phase 4: Slack notified with ↗️ emoji
- [ ] No unexpected errors in any workflow run

### Overall FAIL if ANY of:

- Slash command fails to bump version
- Publish workflow fails
- Rollout not initialized in prod DB within 15 minutes of merge
- `finalize_connector_rollout` API call fails
- `finalize_rollout.yml` workflow fails
- Promote creates a PR with wrong version (e.g., double-bumped)
- Changelog entry missing or has wrong version/PR number
- Job 2 runs during rollback (should be skipped)
- Slack notifications not sent

---

## Troubleshooting

### Rollout not appearing in prod DB

- Check that the publish workflow completed successfully
- Verify the RC metadata marker was created in GCS:
  ```bash
  gsutil ls "gs://prod-airbyte-cloud-connector-metadata-service/metadata/airbyte/source-faker/release_candidate/"
  ```
- Check that the version in `metadata.yaml` contains `-rc.` (the `rc create` step conditions on this)

### Workflow not triggered after finalize API call

- The platform's Temporal workflow dispatches the GitHub workflow via `repository_dispatch`
- Check if the rollout state transitioned correctly in the prod DB
- Check GitHub Actions for any `finalize_rollout` runs (even failed ones)

### Promote creates wrong version

- If the version is double-bumped, this indicates the old double-`bump-version` bug has regressed
- Check that `bump-version --bump-type promote --no-changelog` is called exactly once
- Check that `changelog add` is used for the changelog (not a second `bump-version` call)

### Ops CLI command not found

- The workflow installs via `uv tool install airbyte-internal-ops` (latest from PyPI)
- If `changelog add` is not found, a new release of `airbyte-internal-ops` is needed
- Check: `pip index versions airbyte-internal-ops` and compare to the latest release

## Reference

- **Finalize workflow:** [`finalize_rollout.yml`](https://github.com/airbytehq/airbyte/blob/master/.github/workflows/finalize_rollout.yml)
- **Publish workflow:** [`publish_connectors.yml`](https://github.com/airbytehq/airbyte/blob/master/.github/workflows/publish_connectors.yml)
- **Bump RC slash command:** [`bump-progressive-rollout-version-command.yml`](https://github.com/airbytehq/airbyte/blob/master/.github/workflows/bump-progressive-rollout-version-command.yml)
- **Progressive rollout docs:** [`progressive-rollouts.md`](https://github.com/airbytehq/airbyte-ops-mcp/blob/main/docs/progressive-rollouts.md)
- **Ops CLI source (registry commands):** [`cli/registry.py`](https://github.com/airbytehq/airbyte-ops-mcp/blob/main/src/airbyte_ops_mcp/cli/registry.py)
- **Ops CLI source (local commands):** [`cli/local.py`](https://github.com/airbytehq/airbyte-ops-mcp/blob/main/src/airbyte_ops_mcp/cli/local.py)
