# Connector Registry

The connector registry stores versioned metadata, specs, icons, and documentation for every Airbyte connector. It powers connector discovery in Airbyte Cloud, OSS, and agent-based products.

This document covers the registry implementation in `airbyte-ops-mcp`. For progressive rollout workflows, see [progressive-rollouts.md](progressive-rollouts.md).

## How artifacts reach the registry

The following diagram shows the path from a connector's source tree to the published registry indexes:

```mermaid
flowchart TB
    A[Connector source tree] -->|generate| B[Local artifacts]
    B -->|publish| C["Versioned dir in GCS<br/>(e.g. source-github/1.2.3/)"]
    C -->|compile| D["latest/ dir updated<br/>+ registry indexes rebuilt"]
    D --> E[cloud_registry.json]
    D --> F[oss_registry.json]
```

1. **Generate** reads the connector's source tree and produces artifacts locally.
2. **Publish** uploads versioned artifacts to GCS, skipping unchanged files.
3. **Compile** scans all versioned directories, determines the latest non-yanked version for each connector, syncs the `latest/` pointers, and rebuilds the global index files.

## Store types

Two independent registry stores exist today:

- **CORAL** -- the Airbyte Cloud and OSS connector registry, stored in Google Cloud Storage (GCS). Connector names use the `source-` or `destination-` prefix.
- **SONAR** -- the agent connector registry, stored in Amazon S3. Connector names are bare (for example, `stripe`, `github`). SONAR registry support is planned but not yet launched.

Each store has a dev and prod environment:

```text
CORAL  prod  gs://prod-airbyte-cloud-connector-metadata-service
CORAL  dev   gs://dev-airbyte-cloud-connector-metadata-service-2
SONAR  prod  s3://airbyte-connector-registry
SONAR  dev   s3://airbyte-connector-registry-dev
```

### Store target syntax

CLI commands accept a `--store` flag in the format `<type>:<env>[/<prefix>]`:

```bash
--store coral:dev              # coral dev bucket
--store coral:prod             # coral prod bucket
--store coral:dev/aj-test100   # coral dev bucket with path prefix "aj-test100"
--store sonar:prod             # sonar prod bucket (planned, not yet launched)
```

### Auto-detection

When `--store` is omitted, the tooling infers the correct store from multiple signals:

1. **Connector name** -- names starting with `source-` or `destination-` resolve to CORAL; all others resolve to SONAR.
2. **Working directory** -- the presence of `airbyte-integrations/connectors/` indicates CORAL; `integrations/` alongside `connector-sdk/` indicates SONAR.
3. **Environment variable** -- `AIRBYTE_REGISTRY_STORE` (for example, `coral:dev`).

Explicit sources take priority. If only auto-detected signals are available and they disagree, the command fails with an error.

## GCS bucket layout (CORAL)

All CORAL connector metadata lives under the `metadata/airbyte/` prefix:

```text
gs://prod-airbyte-cloud-connector-metadata-service/
  metadata/
    airbyte/
      source-github/
        1.0.0/
          metadata.yaml
          spec.json
          icon.svg
          doc.md
          manifest.yaml        # declarative connectors only
          components.zip       # declarative connectors only
          components.zip.sha256
        1.0.1/
          ...
        1.1.0-rc.1/
          metadata.yaml
          ...
        latest/
          metadata.yaml        # copy of the latest non-yanked version
          ...
        release_candidate/
          metadata.yaml        # copy of the active RC version, if any
      destination-postgres/
        ...
  registries/
    v0/
      cloud_registry.json      # global index for Cloud connectors
      oss_registry.json        # global index for OSS connectors
  resources/
    connector_stubs/
      v1/
        connector_stubs.json   # enterprise connector placeholder entries
```

### Special directories

- **`latest/`** -- mirrors the artifacts of the highest non-yanked semver version. Updated by `compile`.
- **`release_candidate/`** -- holds a copy of the active RC's `metadata.yaml`. Created by `rc create`, deleted by `rc cleanup`.

## Artifacts

Each published version directory can contain the following files:

| File | Description |
|---|---|
| `metadata.yaml` | Core connector metadata: name, version, Docker image, tags, support level, breaking changes, registry overrides. |
| `spec.json` | Connector specification (JSON Schema of the configuration the user must provide). |
| `icon.svg` | Connector icon. |
| `doc.md` | Connector documentation. |
| `manifest.yaml` | Declarative connector manifest (low-code/manifest-only connectors). |
| `components.zip` | Custom Python components archive (declarative connectors with custom code). |
| `components.zip.sha256` | SHA-256 checksum of the components archive. |
| `version-yank.yml` | Yank marker. Present only for yanked versions. |

## Operations

### Generate

`generate_version_artifacts` produces all version artifacts locally from a connector's source tree. It:

1. Reads `metadata.yaml` from the connector directory.
2. Runs Docker to extract the connector spec (`spec.json`).
3. Copies the icon, documentation, and manifest files.
4. Builds `cloud.json` and `oss.json` registry entries.
5. Writes everything to a local output directory.

```bash
airbyte-ops registry connector-version artifacts generate \
  --metadata-file /path/to/airbyte/airbyte-integrations/connectors/source-github/metadata.yaml \
  --docker-image airbyte/source-github:1.2.3
```

### Publish

`publish_version_artifacts` uploads locally generated artifacts to GCS. It syncs the local artifacts directory to the versioned GCS path, removing stale remote files that no longer exist locally. Pre-publish validation runs automatically before upload.

```bash
airbyte-ops registry connector-version artifacts publish \
  --name source-github --version 1.2.3 \
  --artifacts-dir /path/to/generated/artifacts \
  --store coral:dev
```

`publish_connector_metadata` is a lighter-weight operation that publishes only `metadata.yaml` to a versioned path and optionally updates the `latest/` pointer.

### Compile

`compile_registry` scans all versioned directories in the bucket, determines the latest non-yanked version for each connector, syncs the `latest/` pointers, and rebuilds the global index files (`cloud_registry.json`, `oss_registry.json`).

```bash
airbyte-ops registry store compile --store coral:dev
```

Key behaviors:

- Yanked versions are excluded from "latest" determination.
- Release candidate versions are excluded from "latest" determination.
- The `--with-legacy-migration` flag handles cleanup of disabled connector artifacts from the legacy pipeline.
- The `--with-secrets-mask` flag regenerates `registries/v0/specs_secrets_mask.yaml` by scanning all connector specs for properties marked `airbyte_secret: true`. The resulting YAML file lists every secret property name across all connectors, which the platform uses to redact sensitive fields in logs and API responses.

### Yank and unyank

Yanking marks a version as withdrawn by placing a `version-yank.yml` marker file in the version's directory. Yanked versions are excluded when compile determines the latest version.

Yanking is the standard mechanism for rolling back a bad publish. It replaces the legacy "global version override" (version pinning) approach, which is deprecated.

```bash
# Yank a version
airbyte-ops registry connector-version yank \
  --name source-faker --version 1.2.3 --store coral:dev --reason "Critical bug"

# Unyank (remove the marker)
airbyte-ops registry connector-version unyank \
  --name source-faker --version 1.2.3 --store coral:dev
```

The yank marker contains:

```yaml
yanked: true
yanked_at: "2025-01-15T12:00:00+00:00"
reason: "Critical bug"
```

### Rebuild

`rebuild_registry` re-generates artifacts for all or specified connectors. Supports GCS-native copy and fsspec-based copy strategies.

### Mirror

`mirror` copies an entire registry to a local directory, a different GCS bucket, or an S3 bucket. This is used for testing and cross-store synchronization.

```bash
# Mirror to local filesystem
airbyte-ops registry store mirror --local --source-store coral:prod

# Mirror to local filesystem with explicit output path
airbyte-ops registry store mirror --local --source-store coral:prod \
  --output-path-root /tmp/registry-snapshot

# Mirror to another GCS bucket
airbyte-ops registry store mirror --gcs-bucket my-test-bucket --source-store coral:prod

# Mirror to S3
airbyte-ops registry store mirror --s3-bucket my-s3-bucket --source-store coral:prod
```

### Compare

`compare_stores` compares two registry stores, producing per-connector artifact diffs and global index comparisons. Supports tolerated paths for expected differences.

```bash
airbyte-ops registry store compare \
  --store coral:dev \
  --reference-store coral:prod
```

### Audit

`find_unpublished_connectors` compares local `metadata.yaml` files in the monorepo against what's published in GCS. It reports connectors whose `dockerImageTag` hasn't been published yet. Archived and disabled connectors are skipped, as are RC versions.

## Release candidates

Release candidate versions use the format `X.Y.Z-rc.N` (for example, `1.2.3-rc.1`).

The RC lifecycle:

1. **Publish** -- the RC version is published to GCS like any other version (for example, `1.2.3-rc.1/metadata.yaml`).
2. **`rc create`** -- copies the versioned RC metadata to `release_candidate/metadata.yaml`, signaling the platform that an RC is active.
3. **Progressive rollout** -- the platform gradually migrates customers to the RC. See [progressive-rollouts.md](progressive-rollouts.md).
4. **`rc cleanup`** -- deletes `release_candidate/metadata.yaml`. The versioned directory (for example, `1.2.3-rc.1/`) is preserved as an audit trail.

```bash
# Create RC marker
airbyte-ops registry rc create \
  --name source-github --store coral:prod --repo-path /path/to/airbyte

# Clean up RC marker
airbyte-ops registry rc cleanup \
  --name source-github --store coral:prod --repo-path /path/to/airbyte
```

## Validation

Pre-publish validators run before artifacts are uploaded. The `validate_metadata` function runs all validators against the `data` section of a `metadata.yaml`:

- **Tag format** -- all tags must be `KEY:VALUE` pairs.
- **Language tag** -- at least one `language:<LANG>` tag must be present.
- **Breaking changes** -- major version bumps (`N.0.0`) require a `releases.breakingChanges` entry.
- **Docs path** -- the documentation file must exist on disk.
- **PyPI config** -- PyPI publishing requires a `language:python` or `language:low-code` tag.
- **RC/rollout config** -- RC versions require `releases.rolloutConfiguration.enableProgressiveRollout`. Major-version RCs are not allowed.

## Enterprise connectors

Enterprise connectors are published from the `airbytehq/airbyte-enterprise` repo using the same generate → publish → compile pipeline described above. The same CLI commands and MCP tools apply to both the main `airbytehq/airbyte` monorepo and `airbytehq/airbyte-enterprise`.

### Connector stubs

Connector stubs are placeholder catalog entries for enterprise connectors that direct users to the sales funnel. They are stored at:

```text
gs://prod-airbyte-cloud-connector-metadata-service/resources/connector_stubs/v1/connector_stubs.json
```

Two operations manage stubs:

- **`marketing-stubs sync`** -- uploads the local `connector_stubs.json` to GCS.
- **`marketing-stubs check`** -- compares the local file with what's in GCS and reports differences.

## Authentication

Registry operations require GCS credentials. The tooling resolves credentials in this order:

1. `GCS_CREDENTIALS` environment variable.
2. Application Default Credentials (ADC).

See [CONTRIBUTING.md](../CONTRIBUTING.md) for bucket permission requirements.

## CLI reference

All commands live under `airbyte-ops registry`:

```text
airbyte-ops registry
  connector
    list                              List all connectors in the registry
  connector-version
    list                              List versions for a connector
    next                              Compute next prerelease version tag
    yank                              Mark a version as yanked
    unyank                            Remove yank marker
    metadata
      get                             Read connector metadata from GCS
    artifacts
      generate                        Generate version artifacts locally
      publish                         Publish version artifacts to GCS
  rc
    create                            Create release_candidate/ marker in GCS
    cleanup                           Delete release_candidate/ marker from GCS
  store
    mirror                            Mirror entire registry
    compile                           Compile registry indexes and sync latest/ dirs
    compare                           Compare two registry stores
    delete-dev-latest                 Delete latest/ dirs from a dev store
  marketing-stubs
    sync                              Sync connector_stubs.json to GCS
    check                             Compare local stubs file with GCS
```

Most commands accept a store target flag (for example, `--store coral:dev`). Some commands use different flag names such as `--source-store` (mirror) or `--reference-store` (compare). All registry commands require `GCS_CREDENTIALS`.

## Troubleshooting

_This section is a placeholder. Content will be added as common failure modes and operational procedures are documented._

### Verifying registry state

After a publish or yank, use `compare` to check the affected store against prod or a known-good snapshot:

```bash
airbyte-ops registry store compare \
  --store coral:dev \
  --reference-store coral:prod
```

### Re-running compile

If `latest/` pointers or registry indexes are stale, re-run compile:

```bash
airbyte-ops registry store compile --store coral:prod
```

For a full rebuild including the secrets mask:

```bash
airbyte-ops registry store compile --store coral:prod --with-secrets-mask
```

## MCP tools

The following MCP tools are available for programmatic registry access:

- **`get_connector_registry_entry`** -- read a connector's metadata from the prod registry.
- **`get_connector_registry_spec`** -- read a connector's spec from the prod registry.
- **`list_connectors_in_registry`** -- list all connectors in the prod registry.
- **`list_connector_versions_in_registry`** -- list all versions of a connector in the prod registry.

All MCP tools operate against the production CORAL bucket and require `GCS_CREDENTIALS`.
