---
name: registry-generation-e2e-test
description: Run an end-to-end registry artifact generation test. Generates artifacts for a set of connectors, optionally publishes to GCS, compiles, and compares against the prod registry. Use when validating that the new generation pipeline produces artifacts matching the legacy pipeline.
devin-enabled: true
icon: flask-conical
---

# Registry Generation E2E Test

Run a full end-to-end test of the registry artifact generation pipeline. Validates that locally generated artifacts match (or acceptably differ from) the production registry.

## Prerequisites

- **Repos:** `~/repos/airbyte-ops-mcp` (this repo) and `~/repos/airbyte` (connector metadata source)
- **Docker:** Running and accessible (required for `artifacts generate` to run connector specs)
- **Credentials:** `GCS_CREDENTIALS` or `GCP_GSM_CREDENTIALS` environment variable with a service account JSON key that has access to:
  - Read: `prod-airbyte-cloud-connector-metadata-service` (in GCP project `prod-ab-cloud-proj`)
  - Read/Write: `dev-airbyte-cloud-connector-metadata-service-2` (in GCP project `dev-ab-cloud-proj`)
- If `GCS_CREDENTIALS` is not set but `GCP_GSM_CREDENTIALS` is, export it: `export GCS_CREDENTIALS="$GCP_GSM_CREDENTIALS"`

## Devin Secrets Needed

- `GCP_GSM_CREDENTIALS` — Service account JSON key with GCS bucket access (read prod, read/write dev)

## Modes

This skill supports two modes. **GCS mode is recommended** as it is faster (GCS-to-GCS compare avoids large local downloads) and provides a permanent record of each test run.

### GCS Mode (full pipeline — recommended)

Generates artifacts, publishes them to a datestamped directory in the dev GCS bucket, compiles the store, and compares against prod. Creates a permanent record of the test run. Use this by default.

### Local Mode (no GCS writes)

Generates artifacts locally and compares them against prod by downloading reference artifacts. No files are written to GCS. This mode can work for quick checks but is typically slower than GCS mode because it must download all reference blobs locally.

## Step 1: Select Test Connectors

Choose at least 5 connectors covering multiple types. Use the latest GA version from each connector's `metadata.yaml` in the airbyte monorepo.

**Recommended default set** (covers manifest-only, Python, and Java):

| Connector | Type |
|-----------|------|
| source-faker | Python source |
| source-github | Python source |
| source-google-sheets | Manifest-only source |
| source-notion | Manifest-only source |
| destination-s3 | Java destination |

To resolve the current version for each connector:

```bash
# From ~/repos/airbyte, for each connector:
grep 'dockerImageTag:' airbyte-integrations/connectors/<connector-name>/metadata.yaml | head -1
```

Record each connector name and version. These are your test targets.

## Step 2: Generate Artifacts Locally

For each connector, generate artifacts using the CLI. All commands run from `~/repos/airbyte-ops-mcp`.

```bash
uv run airbyte-ops registry connector-version artifacts generate \
  --metadata-file ~/repos/airbyte/airbyte-integrations/connectors/<connector-name>/metadata.yaml \
  --docker-image airbyte/<connector-name>:<version> \
  --output-dir /tmp/e2e-artifacts/<connector-name> \
  --repo-root ~/repos/airbyte
```

**Failure criteria:** If generation fails for ANY connector, the test fails. Record the error and stop.

**Validation:** After generation, verify the output directory contains:
- `metadata.yaml`
- `cloud.json`
- `oss.json`
- `icon.svg`
- `doc.md`
- `version=<semver>` (zero-byte marker file, e.g. `version=7.0.1`)
- For manifest-only connectors (when present in the source): `manifest.yaml`, `components.zip`, `components.zip.sha256`

Note: The connector spec is embedded within `cloud.json` and `oss.json` — there is no separate `spec.json` file.

## Step 3: Choose Mode and Continue

GCS mode is recommended. Skip to **Step 4** unless you specifically need local-only operation.

### Local Mode: Mirror Prod and Compare Locally

Use the `store mirror` CLI with `--local` to download prod artifacts for the selected connectors:

```bash
uv run airbyte-ops registry store mirror \
  --source-store coral:prod \
  --local \
  --output-path-root /tmp/e2e-reference \
  --connector-name <connector-name-1> \
  --connector-name <connector-name-2> \
  --connector-name <connector-name-3> \
  --connector-name <connector-name-4> \
  --connector-name <connector-name-5>
```

**Failure criteria:** If the mirror fails, the test fails. Record the error and stop.

Then diff each connector's generated artifacts against the mirrored prod artifacts:

```bash
# For each connector and artifact (cloud.json, oss.json, metadata.yaml):
diff /tmp/e2e-artifacts/<connector-name>/cloud.json \
  /tmp/e2e-reference/metadata/airbyte/<connector-name>/<version>/cloud.json
diff /tmp/e2e-artifacts/<connector-name>/oss.json \
  /tmp/e2e-reference/metadata/airbyte/<connector-name>/<version>/oss.json
diff /tmp/e2e-artifacts/<connector-name>/metadata.yaml \
  /tmp/e2e-reference/metadata/airbyte/<connector-name>/<version>/metadata.yaml
```

Skip to **Step 6** after collecting diffs.

### GCS Mode: Publish, Compile, Compare

Continue with Steps 4 and 5 below.

## Step 4: Publish to Dev Store (GCS Mode)

Create a datestamped store ID:

```bash
STORE_ID="$(date +%Y%m%d)-e2e-$(openssl rand -hex 3)"
```

For each connector, publish the generated artifacts:

```bash
uv run airbyte-ops registry connector-version artifacts publish \
  --name <connector-name> \
  --version <version> \
  --artifacts-dir /tmp/e2e-artifacts/<connector-name> \
  --store coral:dev/${STORE_ID}
```

**Failure criteria:** If publish fails for ANY connector, the test fails. Do not attempt workarounds — hard fail and report the CLI error.

## Step 5: Compile and Compare (GCS Mode)

### Compile the store

```bash
uv run airbyte-ops registry store compile --store coral:dev/${STORE_ID}
```

**Failure criteria:** If compile fails, the test fails.

### Verify version marker propagation

After compile, verify that the `version=<semver>` marker file exists in both the versioned directory and the `latest/` directory. The marker should arrive in `latest/` via the recursive copy (not written separately).

```bash
# Check versioned dir has marker
gsutil ls gs://dev-airbyte-cloud-connector-metadata-service-2/${STORE_ID}/metadata/airbyte/<connector-name>/<version>/version=<version>

# Check latest/ dir has marker (arrived via recursive copy)
gsutil ls gs://dev-airbyte-cloud-connector-metadata-service-2/${STORE_ID}/metadata/airbyte/<connector-name>/latest/version=<version>
```

### Verify short-circuit on re-compile

Run compile again. The second compile should detect the marker and short-circuit, reporting `latest_already_current` instead of `latest_updated`.

```bash
uv run airbyte-ops registry store compile --store coral:dev/${STORE_ID}
# Expected: latest_updated=0, latest_already_current=N
```

### Compare against prod

```bash
uv run airbyte-ops registry store compare \
  --store coral:dev/${STORE_ID} \
  --reference-store coral:prod
```

The compare command will report:
- **Tolerated diffs:** The `registry_entry_generated_at` timestamp is tolerated by default (different value expected on every generation).
- **Toleration violations:** Fields where presence or type differs between store and reference — these are warnings.
- **Real diffs:** Any remaining differences in artifact content.
- **Index diffs:** Global registry index differences (expected since the dev store only has the test connectors).

## Step 6: Write Structured Results

Create a results directory and write structured output:

```
/tmp/e2e-results/<STORE_ID or "local">/
  results.json        # Structured pass/fail per connector
  summary.md          # Human-readable summary
  diffs/              # Per-connector diff files
    <connector-name>.cloud.json.diff
    <connector-name>.oss.json.diff
    <connector-name>.metadata.yaml.diff
```

### results.json schema

```json
{
  "mode": "local|gcs",
  "store_id": "<STORE_ID or null>",
  "timestamp": "<ISO-8601>",
  "overall_result": "pass|fail",
  "connectors": [
    {
      "name": "<connector-name>",
      "version": "<version>",
      "type": "<manifest-only|python|java>",
      "steps": {
        "generate": {"status": "pass|fail", "error": null},
        "publish": {"status": "pass|fail|skipped", "error": null},
        "compile": {"status": "pass|fail|skipped", "error": null},
        "compare": {
          "status": "pass|fail|skipped",
          "tolerated_diffs": ["<field paths>"],
          "toleration_violations": ["<field paths>"],
          "real_diffs": {
            "cloud.json": {"keys_only_in_store": [], "keys_only_in_reference": [], "values_differ": []},
            "oss.json": {"keys_only_in_store": [], "keys_only_in_reference": [], "values_differ": []},
            "metadata.yaml": {"differs": true}
          }
        }
      }
    }
  ]
}
```

### summary.md

Write a human-readable markdown table summarizing pass/fail per connector per step, plus a section listing all unique diffs found with concrete examples.

## Step 7: Evaluate Results

**Do NOT declare PASS or FAIL without explicit human sign-off.** Present all diffs to the user one at a time and wait for their judgment on each before proceeding.

**Overall PASS** requires ALL of:
- All connectors generated successfully
- All connectors published successfully (GCS mode)
- Compile succeeded (GCS mode)
- Compare completed without errors (GCS mode)
- All diffs have been reviewed and approved by a human (see below)
- Version marker exists in both versioned and `latest/` directories (GCS mode)
- Re-compile short-circuits correctly (GCS mode)

**Diff review process:**
1. Present each diff category to the user one at a time with concrete examples
2. For each diff, wait for the user to decide: fix it (create a PR) or accept it (document in CONTRIBUTING.md)
3. Only after ALL diffs have been reviewed and resolved can you declare PASS or FAIL
4. Reference [Connector Registry 2.0 Migration Changes](../../../CONTRIBUTING.md#connector-registry-20-migration-changes) for previously accepted differences

**Overall FAIL** if ANY of:
- Any connector fails to generate
- Any connector fails to publish (GCS mode)
- Compile fails (GCS mode)
- Compare finds diffs that the user deems unacceptable and have not yet been fixed
- Version marker missing from `latest/` after compile (GCS mode)

## Step 8: Report Results

Send the user a summary including:

1. **Overall result**: PASS or FAIL (only after human sign-off from Step 7)
2. **Per-connector table**: showing status of each step
3. **Diffs found**: concrete examples of any differences, presented one at a time
4. **Store location** (GCS mode): `gs://dev-airbyte-cloud-connector-metadata-service-2/<STORE_ID>/`
5. **Attach** `results.json` and `summary.md` files

If any new diffs are discovered, file a GitHub issue in `airbytehq/airbyte-ops-mcp` documenting them — use the `create-github-issue` skill. **Issue filing is a one-way process:** log new findings, but do not read or reference existing issues as a source of truth for what is "known" or "acceptable." Your job is to discover and report diffs independently. Only use markdown documents in this repo (e.g., CONTRIBUTING.md) as a reference for previously accepted differences.

## Reference

- **Dev bucket:** `dev-airbyte-cloud-connector-metadata-service-2` (project: `dev-ab-cloud-proj`)
- **Prod bucket:** `prod-airbyte-cloud-connector-metadata-service` (project: `prod-ab-cloud-proj`)
- **Accepted migration differences:** [Connector Registry 2.0 Migration Changes](../../../CONTRIBUTING.md#connector-registry-20-migration-changes)
- **Compare improvements PR:** https://github.com/airbytehq/airbyte-ops-mcp/pull/446
