# Connector Registry

The connector registry stores versioned metadata, specs, icons, and documentation for every Airbyte connector. It powers connector discovery in Airbyte Cloud, OSS, and agent-based products.

This document covers the registry implementation in `airbyte-ops-mcp`. For progressive rollout workflows, see [progressive-rollouts.md](progressive-rollouts.md).

## How artifacts reach the registry

The following diagram shows the path from a connector's source tree to the published registry indexes:

```mermaid
flowchart TB
    A[Connector source tree] -->|generate| B[Local artifacts]
    B -->|publish| C["Versioned dir in GCS<br/>(e.g. source-github/1.2.3/)"]
    C -->|compile| D["latest/ dir updated<br/>+ registry indexes rebuilt"]
    D --> E[cloud_registry.json]
    D --> F[oss_registry.json]
```

1. **Generate** reads the connector's source tree and produces artifacts locally.
2. **Publish** uploads versioned artifacts to GCS, skipping unchanged files.
3. **Compile** scans all versioned directories, determines the latest non-yanked version for each connector, syncs the `latest/` pointers, and rebuilds the global index files.

## Store types

Two independent registry stores exist today:

- **CORAL** -- the Airbyte Cloud and OSS connector registry, stored in Google Cloud Storage (GCS). Connector names use the `source-` or `destination-` prefix.
- **SONAR** -- the agent connector registry, stored in Amazon S3. Connector names are bare (for example, `stripe`, `github`). SONAR registry support is planned but not yet launched.

Each store has a dev and prod environment:

```text
CORAL  prod  gs://prod-airbyte-cloud-connector-metadata-service
CORAL  dev   gs://dev-airbyte-cloud-connector-metadata-service-2
SONAR  prod  s3://airbyte-connector-registry
SONAR  dev   s3://airbyte-connector-registry-dev
```

### Store target syntax

CLI commands accept a `--store` flag in the format `<type>:<env>[/<prefix>]`:

```bash
--store coral:dev              # coral dev bucket
--store coral:prod             # coral prod bucket
--store coral:dev/aj-test100   # coral dev bucket with path prefix "aj-test100"
--store sonar:prod             # sonar prod bucket (planned, not yet launched)
```

### Auto-detection

When `--store` is omitted, the tooling infers the correct store from multiple signals:

1. **Connector name** -- names starting with `source-` or `destination-` resolve to CORAL; all others resolve to SONAR.
2. **Working directory** -- the presence of `airbyte-integrations/connectors/` indicates CORAL; `integrations/` alongside `connector-sdk/` indicates SONAR.
3. **Environment variable** -- `AIRBYTE_REGISTRY_STORE` (for example, `coral:dev`).

Explicit sources take priority. If only auto-detected signals are available and they disagree, the command fails with an error.

## GCS bucket layout (CORAL)

All CORAL connector metadata lives under the `metadata/airbyte/` prefix:

```text
gs://prod-airbyte-cloud-connector-metadata-service/
  metadata/
    airbyte/
      source-github/
        1.0.0/
          metadata.yaml
          spec.json
          icon.svg
          doc.md
          manifest.yaml        # declarative connectors only
          components.zip       # declarative connectors only
          components.zip.sha256
        1.0.1/
          ...
        1.1.0-rc.1/
          metadata.yaml
          ...
        latest/
          metadata.yaml        # copy of the latest non-yanked version
          ...
        release_candidate/
          metadata.yaml        # copy of the active RC version, if any
      destination-postgres/
        ...
  registries/
    v0/
      cloud_registry.json      # global index for Cloud connectors
      oss_registry.json        # global index for OSS connectors
  resources/
    connector_stubs/
      v1/
        connector_stubs.json   # enterprise connector placeholder entries
```

### Special directories

- **`latest/`** -- mirrors the artifacts of the highest non-yanked semver version. Updated by `compile`.
- **`release_candidate/`** -- holds a copy of the active RC's `metadata.yaml`. Created by `rc create`, deleted by `rc cleanup`.

## Artifacts

Each published version directory can contain the following files:

| File | Description |
|---|---|
| `metadata.yaml` | Core connector metadata: name, version, Docker image, tags, support level, breaking changes, registry overrides. |
| `spec.json` | Connector specification (JSON Schema of the configuration the user must provide). |
| `icon.svg` | Connector icon. |
| `doc.md` | Connector documentation. |
| `manifest.yaml` | Declarative connector manifest (low-code/manifest-only connectors). |
| `components.zip` | Custom Python components archive (declarative connectors with custom code). |
| `components.zip.sha256` | SHA-256 checksum of the components archive. |
| `version-yank.yml` | Yank marker. Present only for yanked versions. |

## Adjacent connector metrics export

Connector quality metrics are not versioned registry artifacts, but they are adjacent to the registry because Airbyte Docs and other downstream consumers have historically read them from registry entries at `generated.metrics`.

The metrics source of truth is the BigQuery table `airbyte-data-prod.airbyte_warehouse_reporting.connector_quality_metrics`, defined in the `airbyte-warehouse` dbt model `dbt_core/models/4-reporting/engineering/connector_quality_metrics.sql`. The table contains one row per connector, platform, and version scope, with categorical `sync_success_rate` and `usage` values.

An internal Airbyte Cloud connection exports that table to GCS:

- Connection name: `connector_quality_metrics → Google Cloud Storage (GCS)`.
- Connection ID: [`6cf642d8-f259-4f48-8ed4-fa371d709eb6`](https://cloud.airbyte.com/workspaces/bed3b473-1518-4461-a37f-730ea3d3a848/connections/6cf642d8-f259-4f48-8ed4-fa371d709eb6).
- Source: `BigQuery (ab-analytics)`, stream `connector_quality_metrics`.
- Destination: `Google Cloud Storage (GCS) - prod-ab-cloud-proj`.
- Schedule: cron `0 0 23 * * ?` in UTC.
- Output prefix: [`gs://ab-analytics-connector-metrics/data/connector_quality_metrics/`](https://console.cloud.google.com/storage/browser/ab-analytics-connector-metrics/data/connector_quality_metrics;tab=objects?project=ab-analytics-308500).
- Output format: uncompressed JSONL, with filenames like `YYYY_MM_DD_<epoch>_0.jsonl`.

The legacy metadata-service Dagster registry pipeline read the latest JSONL blob from that prefix and attached metrics to registry entries under `generated.metrics`. Consumers that need the latest metrics should list `*.jsonl` objects under the prefix and select the lexicographically latest filename, which matches the legacy reader behavior.

The JSONL export provides one row per connector, version scope, and platform. The historical registry rendering stored those rows under `generated.metrics` by platform. See https://github.com/airbytehq/airbyte-ops-mcp/issues/720 for the current status of adding metrics back to registry compilation. Historical registry shape:

```json
{
  "generated": {
    "metrics": {
      "all": {
        "sync_success_rate": "high",
        "usage": "high"
      },
      "cloud": {
        "sync_success_rate": "high",
        "usage": "medium"
      },
      "oss": {
        "sync_success_rate": "medium",
        "usage": "low"
      }
    }
  }
}
```

## Operations

### Generate

`generate_version_artifacts` produces all version artifacts locally from a connector's source tree. It:

1. Reads `metadata.yaml` from the connector directory.
2. Runs Docker to extract the connector spec (`spec.json`).
3. Copies the icon, documentation, and manifest files.
4. Builds `cloud.json` and `oss.json` registry entries.
5. Writes everything to a local output directory.

```bash
airbyte-ops registry connector-version artifacts generate \
  --metadata-file /path/to/airbyte/airbyte-integrations/connectors/source-github/metadata.yaml \
  --docker-image airbyte/source-github:1.2.3
```

### Publish

`publish_version_artifacts` uploads locally generated artifacts to GCS. It syncs the local artifacts directory to the versioned GCS path, removing stale remote files that no longer exist locally. Pre-publish validation runs automatically before upload.

```bash
airbyte-ops registry connector-version artifacts publish \
  --name source-github --version 1.2.3 \
  --artifacts-dir /path/to/generated/artifacts \
  --store coral:dev
```

`publish_connector_metadata` is a lighter-weight operation that publishes only `metadata.yaml` to a versioned path and optionally updates the `latest/` pointer.

### Compile

`compile_registry` scans all versioned directories in the bucket, determines the latest non-yanked version for each connector, syncs the `latest/` pointers, and rebuilds the global index files (`cloud_registry.json`, `oss_registry.json`).

```bash
airbyte-ops registry store compile --store coral:dev
```

Key behaviors:

- Yanked versions are excluded from "latest" determination.
- Release candidate versions are excluded from "latest" determination.
- The `--with-legacy-migration` flag handles cleanup of disabled connector artifacts from the legacy pipeline.
- The `--with-secrets-mask` flag regenerates `registries/v0/specs_secrets_mask.yaml` by scanning all connector specs for properties marked `airbyte_secret: true`. The resulting YAML file lists every secret property name across all connectors, which the platform uses to redact sensitive fields in logs and API responses.

### Yank and unyank

Yanking marks a version as withdrawn by placing a `version-yank.yml` marker file in the version's directory. Yanked versions are excluded when compile determines the latest version.

Yanking is the standard mechanism for rolling back a bad publish. It replaces the legacy "global version override" (version pinning) approach, which is deprecated.

```bash
# Yank a version
airbyte-ops registry connector-version yank \
  --name source-faker --version 1.2.3 --store coral:dev --reason "Critical bug"

# Unyank (remove the marker)
airbyte-ops registry connector-version unyank \
  --name source-faker --version 1.2.3 --store coral:dev
```

The yank marker contains:

```yaml
yanked: true
yanked_at: "2025-01-15T12:00:00+00:00"
reason: "Critical bug"
```

### Rebuild

`rebuild_registry` re-generates artifacts for all or specified connectors. Supports GCS-native copy and fsspec-based copy strategies.

### Mirror

`mirror` copies an entire registry to a local directory, a different GCS bucket, or an S3 bucket. This is used for testing and cross-store synchronization.

```bash
# Mirror to local filesystem
airbyte-ops registry store mirror --local --source-store coral:prod

# Mirror to local filesystem with explicit output path
airbyte-ops registry store mirror --local --source-store coral:prod \
  --output-path-root /tmp/registry-snapshot

# Mirror to another GCS bucket
airbyte-ops registry store mirror --gcs-bucket my-test-bucket --source-store coral:prod

# Mirror to S3
airbyte-ops registry store mirror --s3-bucket my-s3-bucket --source-store coral:prod
```

### Compare

`compare_stores` compares two registry stores, producing per-connector artifact diffs and global index comparisons. Supports tolerated paths for expected differences.

```bash
airbyte-ops registry store compare \
  --store coral:dev \
  --reference-store coral:prod
```

### Audit

`find_unpublished_connectors` compares local `metadata.yaml` files in the monorepo against what's published in GCS. It reports connectors whose `dockerImageTag` hasn't been published yet. Archived and disabled connectors are skipped, as are RC versions.

## Release candidates

Release candidate versions use the format `X.Y.Z-rc.N` (for example, `1.2.3-rc.1`).

The RC lifecycle:

1. **Publish** -- the RC version is published to GCS like any other version (for example, `1.2.3-rc.1/metadata.yaml`).
2. **`rc create`** -- copies the versioned RC metadata to `release_candidate/metadata.yaml`, signaling the platform that an RC is active.
3. **Progressive rollout** -- the platform gradually migrates customers to the RC. See [progressive-rollouts.md](progressive-rollouts.md).
4. **`rc cleanup`** -- deletes `release_candidate/metadata.yaml`. The versioned directory (for example, `1.2.3-rc.1/`) is preserved as an audit trail.

```bash
# Create RC marker
airbyte-ops registry rc create \
  --name source-github --store coral:prod --repo-path /path/to/airbyte

# Clean up RC marker
airbyte-ops registry rc cleanup \
  --name source-github --store coral:prod --repo-path /path/to/airbyte
```

## Validation

Pre-publish validators run before artifacts are uploaded. The `validate_metadata` function runs all validators against the `data` section of a `metadata.yaml`:

- **Tag format** -- all tags must be `KEY:VALUE` pairs.
- **Language tag** -- at least one `language:<LANG>` tag must be present.
- **Breaking changes** -- major version bumps (`N.0.0`) require a `releases.breakingChanges` entry.
- **Docs path** -- the documentation file must exist on disk.
- **PyPI config** -- PyPI publishing requires a `language:python` or `language:low-code` tag.
- **RC/rollout config** -- RC versions require `releases.rolloutConfiguration.enableProgressiveRollout`. Major-version RCs are not allowed.

## Enterprise connectors

Enterprise connectors are published from the `airbytehq/airbyte-enterprise` repo using the same generate → publish → compile pipeline described above. The same CLI commands and MCP tools apply to both the main `airbytehq/airbyte` monorepo and `airbytehq/airbyte-enterprise`.

### Connector stubs

Connector stubs are placeholder catalog entries for enterprise connectors that direct users to the sales funnel. They are stored at:

```text
gs://prod-airbyte-cloud-connector-metadata-service/resources/connector_stubs/v1/connector_stubs.json
```

Two operations manage stubs:

- **`marketing-stubs sync`** -- uploads the local `connector_stubs.json` to GCS.
- **`marketing-stubs check`** -- compares the local file with what's in GCS and reports differences.

## Authentication

Registry operations require GCS credentials. The tooling resolves credentials in this order:

1. `GCS_CREDENTIALS` environment variable.
2. Application Default Credentials (ADC).

See [CONTRIBUTING.md](../CONTRIBUTING.md) for bucket permission requirements.

## CLI reference

All commands live under `airbyte-ops registry`:

```text
airbyte-ops registry
  connector
    list                              List all connectors in the registry
  connector-version
    list                              List versions for a connector
    next                              Compute next prerelease version tag
    yank                              Mark a version as yanked
    unyank                            Remove yank marker
    metadata
      get                             Read connector metadata from GCS
    artifacts
      generate                        Generate version artifacts locally
      publish                         Publish version artifacts to GCS
  rc
    create                            Create release_candidate/ marker in GCS
    cleanup                           Delete release_candidate/ marker from GCS
  store
    mirror                            Mirror entire registry
    compile                           Compile registry indexes and sync latest/ dirs
    compare                           Compare two registry stores
    delete-dev-latest                 Delete latest/ dirs from a dev store
  marketing-stubs
    sync                              Sync connector_stubs.json to GCS
    check                             Compare local stubs file with GCS
```

Most commands accept a store target flag (for example, `--store coral:dev`). Some commands use different flag names such as `--source-store` (mirror) or `--reference-store` (compare). All registry commands require `GCS_CREDENTIALS`.

## Troubleshooting

_This section is a placeholder. Content will be added as common failure modes and operational procedures are documented._

### Verifying registry state

After a publish or yank, use `compare` to check the affected store against prod or a known-good snapshot:

```bash
airbyte-ops registry store compare \
  --store coral:dev \
  --reference-store coral:prod
```

### Re-running compile

If `latest/` pointers or registry indexes are stale, re-run compile:

```bash
airbyte-ops registry store compile --store coral:prod
```

For a full rebuild including the secrets mask:

```bash
airbyte-ops registry store compile --store coral:prod --with-secrets-mask
```

## MCP tools

The following MCP tools are available for programmatic registry access:

- **`get_connector_registry_entry`** -- read a connector's metadata from the prod registry.
- **`get_connector_registry_spec`** -- read a connector's spec from the prod registry.
- **`list_connectors_in_registry`** -- list all connectors in the prod registry.
- **`list_connector_versions_in_registry`** -- list all versions of a connector in the prod registry.

All MCP tools operate against the production CORAL bucket and require `GCS_CREDENTIALS`.
