---
name: progressive-rollout-e2e-test
description: End-to-end test of the connector progressive rollout lifecycle using bot-safe MCP tools, publish workflows, prod rollout DB checks, rollback, second candidate version, and promote.
devin-enabled: true
icon: flask-conical
---

# Progressive Rollout E2E Test

Run a full end-to-end test of the connector progressive rollout lifecycle. This test exercises candidate version creation, publish, platform initialization, rollback, second candidate version creation, and promotion.

For Devin sessions, prefer `airbyte-ops-mcp` MCP tools over GitHub slash-command comments. Use `bump_version_in_repo` for version bumps, `publish_connector_to_airbyte_registry` to trigger the connector publish/registry workflow, `trigger_ci_workflow` only for general workflow-dispatch cases, and `check_ci_workflow_status` or PR checks for workflow status. Bot comments like `/bump-version` or `/publish-connectors-prerelease` may not be processed.

This is a human-in-the-loop test operation. Devin can prepare branches, draft PRs, tool calls, and evidence, but a human must approve or perform production-impacting gates. Do not treat this as a fully autonomous workflow.

## Agent Secrets Needed

- `GCP_GSM_CREDENTIALS` — service account JSON key with GCS bucket access. Registry tools can use this when `GCS_CREDENTIALS` is not set.
- `GCP_PROD_DB_ACCESS_CREDENTIALS` — service account JSON key for prod DB replica access.
- `AIRBYTE_CLOUD_CLIENT_ID` — Airbyte Cloud API credential for finalize operations.
- `AIRBYTE_CLOUD_CLIENT_SECRET` — Airbyte Cloud API credential for finalize operations.
- `AIRBYTE_INTERNAL_ADMIN_FLAG=airbyte.io` — non-secret env flag required for `finalize_connector_rollout` calls.
- `GITHUB_CI_WORKFLOW_TRIGGER_PAT` — used by ops MCP workflow-dispatch helpers.

## Repos and setup

- Airbyte monorepo: `~/repos/airbyte`
- Ops MCP repo: `~/repos/airbyte-ops-mcp`
- Read `AGENTS.md` and `CONTRIBUTING.md` when present.
- Use `poe mcp-tool-test <tool_name> '<json_args>'` from `~/repos/airbyte-ops-mcp` when testing local MCP tool behavior.
- Start Cloud SQL Proxy before prod DB query tools:

```bash
cd ~/repos/airbyte-ops-mcp
uv run airbyte-ops cloud db start-proxy --port 15432
export USE_CLOUD_SQL_PROXY=1
export DB_PORT=15432
```

## Test connector default

Use `source-faker` unless the user specifies another connector:

- Connector name: `source-faker`
- Docker repository: `airbyte/source-faker`
- Actor definition ID: `dfd88b22-b603-4c3d-aad7-3701784586b1`

Always read the current stable version from `airbyte-integrations/connectors/<connector>/metadata.yaml` before starting. Do not hard-code the current version from this skill.

## Choose a test modality

Pick one modality before starting and use it consistently in both candidate-version phases:

1. **RC pre-release modality**: use `bump_version_in_repo` with `bump_type: "rc"` for candidate 1 and candidate 2. Expected versions are `RC1_VERSION` and `RC2_VERSION`, for example `7.2.0-rc.1` and `7.2.0-rc.2`. This validates the pre-release/RC code path.
2. **Subsequent patch-release modality**: use `bump_version_in_repo` with `bump_type: "patch"` for candidate 1 and candidate 2, then explicitly set `releases.rolloutConfiguration.enableProgressiveRollout: true` in each test PR if the patch bump did not set it. Expected versions are `PATCH1_VERSION` and `PATCH2_VERSION`, for example `7.1.2` and `7.1.3`. Do not use `rc`, `patch_rc`, `minor_rc`, `major_rc`, or `promote`; this modality intentionally avoids the pre-release/RC code path.

Unless the requester specifies the patch-release modality, use the RC pre-release modality.

## Human-in-the-loop approval gates

Stop at these gates unless the user explicitly approves proceeding:

1. Merging any test PR. In this test, assume Devin cannot merge PRs by itself. Ask the requester or parent session to approve or perform the merge, then continue from the merged commit.
2. Any rollback, promote, or finalize call.

For finalize calls, first request Slack approval with `escalate_to_human` and pass the returned human Slack approval URL as `approval_comment_url`. Do not use a GitHub URL or a bot-authored approval comment. If a Slack approval URL is not available, stop and report the untested phase instead of attempting rollback, promote, or finalize.

## Step 0: Collect finalize approval URL

Before any rollback, promote, or finalize call, collect a Slack approval URL with `escalate_to_human`:

```bash
cd ~/repos/airbyte-ops-mcp
uv run poe mcp-tool-test escalate_to_human '{
  "target_person": "<USER_EMAIL>",
  "message": "Requesting approval for progressive rollout E2E finalize operations on source-faker. This will roll back one candidate version and promote a second candidate version.",
  "agent_session_url": "<DEVIN_SESSION_URL>",
  "approval_requested": true,
  "approval_request_summary": "Progressive rollout E2E test: roll back candidate 1 and promote candidate 2 for source-faker",
  "connector_name": "source-faker",
  "request_type": "approval"
}'
```

Wait for the human to click "Approve" in Slack. Record the resulting Slack URL as `APPROVAL_COMMENT_URL`. Reuse that URL for Phase 2 and Phase 4 finalize calls. If no Slack approval URL is available, stop before finalize operations and report those phases as untested.

## Phase 1: Create candidate 1

1. Create a fresh branch off `master` in `~/repos/airbyte`.
2. Make a small connector-local test marker change so connector CI is triggered.
3. Create the draft PR with `git_create_pr` so you have a real PR number. Do not mark it ready.
4. Use `bump_version_in_repo` instead of a `/bump-version` comment. `repo_path` must be an absolute path. Replace `/absolute/path/to/repos/airbyte` with `realpath ~/repos/airbyte`, or with the absolute repo path for the current environment. Use the selected modality's bump type (`"rc"` or `"patch"`). Include `pr_number` so the docs changelog does not get a `*PR_NUMBER_PLACEHOLDER*` link:

```bash
cd ~/repos/airbyte-ops-mcp
uv run poe mcp-tool-test bump_version_in_repo '{
  "repo_path": "/absolute/path/to/repos/airbyte",
  "connector_name": "source-faker",
  "bump_type": "rc",
  "changelog_message": "Progressive rollout e2e test",
  "pr_number": 12345,
  "dry_run": false
}'
```

5. For patch-release modality only, verify `releases.rolloutConfiguration.enableProgressiveRollout: true` is set. If the patch bump did not set it, edit `metadata.yaml` in the same PR to enable it.
6. Commit and push the version-bump changes to the draft PR branch.
7. Verify before merge. Candidate bumps should not add prior-version `registryOverrides.*.dockerImageTag` pins; the compiled registry keeps the previous GA version as the default and exposes the candidate separately under rollout metadata.
   - PR is draft.
   - RC modality: `metadata.yaml` has `dockerImageTag: <NEXT_MINOR>-rc.1`.
   - Patch modality: `metadata.yaml` has `dockerImageTag: <CURRENT_PATCH_PLUS_1>` with no `-rc` suffix.
   - No new `registryOverrides.cloud.dockerImageTag` or `registryOverrides.oss.dockerImageTag` pins were added for the candidate.
   - `releases.rolloutConfiguration.enableProgressiveRollout: true`.
   - `pyproject.toml` matches the candidate version when present.
   - Docs changelog contains the candidate-version row with the actual draft PR number, not `*PR_NUMBER_PLACEHOLDER*`.
8. Wait for PR CI to complete. Do not merge. Ask the human requester or parent session to approve the PR, mark it ready, and perform the merge before moving to publish verification.

## Phase 1 publish verification after merge approval

After the human requester or parent session explicitly approves or performs the merge:

1. Confirm the PR was merged to `master`.
2. If the merge did not automatically publish the connector, trigger the connector publish workflow with `publish_connector_to_airbyte_registry`. For general CI dispatches, use `trigger_ci_workflow`.
3. Monitor `publish_connectors.yml` using `check_ci_workflow_status` or PR checks.
4. Verify compiled registry:

```bash
curl -s "https://connectors.airbyte.com/files/registries/v0/cloud_registry.json" | python3 - <<'PY'
import json, sys
registry = json.load(sys.stdin)
for source in registry.get("sources", []):
    if source.get("dockerRepository") == "airbyte/source-faker":
        print("GA dockerImageTag:", source.get("dockerImageTag"))
        print("releaseCandidates:", source.get("releases", {}).get("releaseCandidates", {}))
        break
PY
```

Expected:
- Top-level GA `dockerImageTag` remains the previous stable version.
- `releases.releaseCandidates` contains the candidate version.
- The candidate entry has `dockerImageTag` equal to the selected modality's candidate version.
- The candidate entry has `enableProgressiveRollout: true`.

5. Verify the release candidate marker:

```bash
curl -s -o /dev/null -w "%{http_code}" \
  "https://connectors.airbyte.com/files/metadata/airbyte/source-faker/release_candidate/cloud.json"
```

Expected: `200`.

6. Verify prod rollout DB:

```bash
cd ~/repos/airbyte-ops-mcp
export USE_CLOUD_SQL_PROXY=1
export DB_PORT=15432
uv run poe mcp-tool-test query_prod_connector_rollouts '{
  "actor_definition_id": "dfd88b22-b603-4c3d-aad7-3701784586b1",
  "active_only": true
}'
```

Expected:
- Rollout `state` is `initialized`.
- `rc_docker_image_tag` equals the candidate version.
- `rc_docker_repository` equals `airbyte/source-faker`.

Record `ROLLOUT1_ID`.

## Phase 2: Roll back candidate 1

Only after Slack approval from `escalate_to_human`, call `finalize_connector_rollout` with `state: "failed_rolled_back"`:

```bash
uv run poe mcp-tool-test finalize_connector_rollout '{
  "docker_repository": "airbyte/source-faker",
  "docker_image_tag": "<RC1_VERSION_OR_PATCH1_VERSION>",
  "actor_definition_id": "dfd88b22-b603-4c3d-aad7-3701784586b1",
  "rollout_id": "<ROLLOUT1_ID>",
  "state": "failed_rolled_back",
  "approval_comment_url": "<APPROVAL_COMMENT_URL>",
  "error_msg": "Progressive rollout e2e test rollback",
  "failed_reason": "FOUND_ISSUES"
}'
```

Expected:
- `finalize_rollout.yml` runs with rollback action.
- Registry cleanup job runs.
- Promotion PR creation job is skipped.
- Release candidate marker returns `404` after cleanup.
- Compiled registry no longer lists this candidate under `releaseCandidates`.
- Prod DB rollout state is `failed_rolled_back`.

## Phase 3: Create candidate 2

1. Create a new fresh branch off updated `master`.
2. Make another connector-local test marker change.
3. Create a second draft PR and record the PR number.
4. Use `bump_version_in_repo` with the selected modality's bump type (`"rc"` or `"patch"`) and the new `pr_number`, then commit and push the version-bump changes.
5. Verify the new version matches the selected modality: RC modality uses `<RC_BASE>-rc.2`; patch modality uses the next patch version with no `-rc` suffix. No candidate-specific registry override pins should be added, and the changelog row must use the actual second PR number.
6. Do not merge. Ask the human requester or parent session to approve the PR, mark it ready, and perform the merge.
7. After the human-approved merge, repeat publish, registry marker, and DB initialization checks. Record `ROLLOUT2_ID`.

## Phase 4: Promote candidate 2

Only after Slack approval from `escalate_to_human`, call `finalize_connector_rollout` with `state: "succeeded"`:

```bash
uv run poe mcp-tool-test finalize_connector_rollout '{
  "docker_repository": "airbyte/source-faker",
  "docker_image_tag": "<RC2_VERSION_OR_PATCH2_VERSION>",
  "actor_definition_id": "dfd88b22-b603-4c3d-aad7-3701784586b1",
  "rollout_id": "<ROLLOUT2_ID>",
  "state": "succeeded",
  "approval_comment_url": "<APPROVAL_COMMENT_URL>"
}'
```

Expected:
- `finalize_rollout.yml` runs with promote action.
- Registry cleanup job deletes the `release_candidate/` marker and recompiles registry.
- Promotion PR creation job runs.
- A merged PR titled like `chore: finalize promote for source-faker` exists.
- `metadata.yaml` on `master` has the final GA version with no `-rc.N` suffix.
- `enableProgressiveRollout` is `false`.
- No candidate-specific Cloud or OSS `registryOverrides.*.dockerImageTag` pins were added or preserved by the rollout flow.
- Changelog contains the GA promotion entry.
- Prod DB rollout state is `succeeded`.

## Pass/fail criteria

Overall pass requires all of:

- Candidate 1 draft PR created and CI green.
- Candidate 1 published and initialized in prod DB.
- Compiled registry exposes the candidate under `releaseCandidates` while the top-level default remains the previous GA version.
- Rollback workflow runs cleanup and skips promotion PR creation.
- Candidate 2 published and initialized in prod DB.
- Promote workflow creates/merges the GA promotion PR.
- Final master metadata has clean GA version, progressive rollout disabled, and no candidate-specific override pins.
- No unexpected workflow errors.

Overall fail if any of:

- Publish workflow fails or release-block logic unexpectedly blocks the connector.
- `releaseCandidates` is missing after publish and compile.
- The default registry tag changes to the RC instead of the stable version during rollout.
- Rollout is not initialized in prod DB within the expected publish window.
- `finalize_connector_rollout` fails after valid Slack approval.
- Rollback runs the promotion PR job.
- Promote creates the wrong version or stale rollout metadata.
- Slack approval/notification requirements are not met.

## Reporting

For test reports and GitHub PR comments, include:

- PR URLs for RC1, RC2, and promotion PR.
- Workflow URLs and final statuses.
- Registry JSON excerpts showing stable default and RC entries.
- RC marker HTTP status evidence.
- Prod DB rollout IDs and states.
- Approval gates and any untested phases.
- Devin session URL.

This is shell/MCP/GitHub Actions testing, so do not record a desktop video unless a browser UI becomes part of the scenario.
