# Ops Webapp and MCP infrastructure bootstrap

This Pulumi project provisions the Airbyte Ops Webapp at
`https://ops.internal.airbyte.ai` and the internal hosted MCP servers at
`https://mcp.internal.airbyte.ai`.

## Scope

Pulumi owns:

- Cloud Run services: `ops-webapp` (production) and `ops-webapp-preview` (PR previews)
- Cloud Run services: `ops-mcp`, `ops-mcp-preview`, `cloud-mcp`, and `cloud-mcp-preview`
- Runtime service account and Cloud Run/load balancer IAM
- External HTTP(S) load balancer with host-based and path-based routing, managed TLS certificates, and HTTP to HTTPS redirect
- DNS A records for `ops.internal.airbyte.ai`, `preview.ops.internal.airbyte.ai`, and `mcp.internal.airbyte.ai` in the existing `internal-airbyte-ai` Cloud DNS zone

Bootstrap owns:

- Artifact Registry repositories: `us-west3-docker.pkg.dev/internal-airbyte-ai-infra/ops-webapp`, `.../ops-mcp`, and `.../cloud-mcp`
- Bootstrap images: `ops-webapp/ops-webapp:latest`, `ops-mcp/ops-mcp:latest`, and `cloud-mcp/cloud-mcp:latest`
- Secret Manager secret containers, values, and runtime access grants for all
  OAuth clients (`ops-webapp-oauth-client-secret`, `ops-webapp-google-oauth-client-secret`,
  `ops-mcp-oauth-client-secret`, `cloud-mcp-oauth-client-secret`)

Pulumi does **not** own Keycloak/OAuth client configuration. The Ops Webapp and
MCP Keycloak clients are managed manually. See
[`docs/keycloak-client-admin.md`](../../docs/keycloak-client-admin.md) for admin
console access and redirect URI updates.

## Stage 0: first apply blockers

The runtime infrastructure lives in `internal-airbyte-ai-infra` and the
`internal.airbyte.ai` DNS zone lives in `airbyte-intranet`.

These are the prerequisites that can block the first full `pulumi up` from
succeeding:

Set local convenience variables:

```bash
export PROJECT="internal-airbyte-ai-infra"
export REGION="us-west3"
export RUNTIME_SA="ops-webapp-sa@internal-airbyte-ai-infra.iam.gserviceaccount.com"
export REPO_ROOT="/path/to/airbyte-ops-mcp"
export OAUTH_CLIENT_SECRET_FILE="/path/to/ops-webapp-oauth-client-secret.txt"
export OPS_MCP_OAUTH_SECRET_FILE="/path/to/ops-mcp-oauth-client-secret.txt"
export CLOUD_MCP_OAUTH_SECRET_FILE="/path/to/cloud-mcp-oauth-client-secret.txt"
```

1. The applying identity can administer resources in
   `internal-airbyte-ai-infra`, including enabling required Google APIs.

   ```bash
   gcloud services enable cloudresourcemanager.googleapis.com \
     --project="$PROJECT"
   ```

2. The same applying identity can add records to the `internal-airbyte-ai` DNS
   zone in `airbyte-intranet`.

   ```bash
   gcloud projects add-iam-policy-binding airbyte-intranet \
     --member=serviceAccount:github-actions-ci@internal-airbyte-ai-infra.iam.gserviceaccount.com \
     --role=roles/dns.admin
   ```

3. The `internal-airbyte-ai` DNS zone exists in `airbyte-intranet`, and there is
   no conflicting `ops.internal.airbyte.ai.` A record.
4. The Artifact Registry repository exists before pushing the bootstrap image.
   If it already exists, leave it in place and skip this command:

   ```bash
   gcloud artifacts repositories create ops-webapp \
     --project="$PROJECT" \
     --location="$REGION" \
     --repository-format=docker \
     --description="Container images for the Airbyte Ops Webapp"
   ```

5. A bootstrap image exists before Cloud Run is created:

   ```bash
   gcloud builds submit "$REPO_ROOT" \
     --project="$PROJECT" \
     --config="$REPO_ROOT/infra/ops-webapp/cloudbuild.bootstrap.yaml" \
     --substitutions=_IMAGE="${REGION}-docker.pkg.dev/${PROJECT}/ops-webapp/ops-webapp:latest"
   ```

6. The OAuth client secret container, runtime access grant, and at least one
   secret version exist before Cloud Run is created:

   ```bash
   gcloud secrets describe ops-webapp-oauth-client-secret --project="$PROJECT" >/dev/null 2>&1 || \
     gcloud secrets create ops-webapp-oauth-client-secret \
       --project="$PROJECT" \
       --replication-policy=automatic

   gcloud secrets add-iam-policy-binding ops-webapp-oauth-client-secret \
     --project="$PROJECT" \
     --member="serviceAccount:${RUNTIME_SA}" \
     --role="roles/secretmanager.secretAccessor"

   gcloud secrets versions add ops-webapp-oauth-client-secret \
     --project="$PROJECT" \
     --data-file="$OAUTH_CLIENT_SECRET_FILE"
   ```

The secret container, secret value, and secret IAM policy are external bootstrap
resources. Pulumi reads the secret name for Cloud Run configuration but does not
create the secret or manage its IAM policy. The signed-in user's OAuth bearer
token is used for Config API requests.

7. The Google OAuth client for BigQuery user-token access. This enables the
   webapp to query BigQuery using each user's own Google identity (rather than a
   shared service account), providing per-user audit trails and leveraging
   existing IAM roles.

   **Project**: `internal-airbyte-ai-infra` (same as the Keycloak OAuth client).

   **Enable required APIs**:

   ```bash
   gcloud services enable bigquery.googleapis.com \
     --project="$PROJECT"
   ```

   **Configure OAuth consent screen and create client** (one-time, Console
   only — `gcloud` does not support standard web-app OAuth client management):

   Step 1 — Consent screen:
   https://console.cloud.google.com/apis/credentials/consent?project=internal-airbyte-ai-infra

   - User type: **Internal** (restricts to `@airbyte.io`, no Google app review)
   - App name: `Airbyte Ops Webapp`
   - Support email: `aj@airbyte.io`
   - Scopes: `openid`, `email`, `profile`,
     `https://www.googleapis.com/auth/bigquery.readonly`

   Step 2 — Create OAuth client ID:
   https://console.cloud.google.com/apis/credentials?project=internal-airbyte-ai-infra

   - Create Credentials → OAuth client ID → **Web application**
   - Name: `ops-webapp-google-oauth`
   - Authorized redirect URIs:
     - `https://ops.internal.airbyte.ai/google-oauth/callback`
     - `https://preview.ops.internal.airbyte.ai/google-oauth/callback`
     - `http://localhost:3000/google-oauth/callback`

   After creation, note the **Client ID** and **Client Secret** values.

   **Store the client secret in Secret Manager**:

   ```bash
   export GOOGLE_OAUTH_CLIENT_SECRET_FILE="/path/to/google-oauth-client-secret.txt"

   # Create the secret container (one-time)
   gcloud secrets create ops-webapp-google-oauth-client-secret \
     --project="$PROJECT" \
     --replication-policy=automatic

   # Grant the runtime SA access
   gcloud secrets add-iam-policy-binding ops-webapp-google-oauth-client-secret \
     --project="$PROJECT" \
     --member="serviceAccount:${RUNTIME_SA}" \
     --role="roles/secretmanager.secretAccessor"

   # Add the secret value
   gcloud secrets versions add ops-webapp-google-oauth-client-secret \
     --project="$PROJECT" \
     --data-file="$GOOGLE_OAUTH_CLIENT_SECRET_FILE"
   ```

   > **Note:** The Google OAuth Client ID is hardcoded in the app source
   > (`google_oauth.py`) because it is public (visible in every OAuth redirect
   > URL). No Secret Manager entry or env var is needed for it.

   **Environment variable** consumed by the webapp at runtime:

   - `AIRBYTE_OPS_WEBAPP_GOOGLE_CLIENT_SECRET` — Google OAuth Client Secret

   Wired to Cloud Run via Pulumi (`infra/ops-webapp/__main__.py`).

   **Scopes requested during the OAuth flow**:

   - `openid` — required for ID token
   - `email` — user email claim
   - `profile` — user display name
   - `https://www.googleapis.com/auth/bigquery.readonly` — read-only BigQuery access

   **User IAM requirements**: Users must have `roles/bigquery.jobUser` (or
   equivalent) on `airbyte-data-prod` through their Google Workspace IAM
   membership. No per-user grants should be needed for `@airbyte.io` accounts
   that already have BigQuery access. Verify with:

   ```bash
   gcloud projects get-iam-policy airbyte-data-prod \
     --flatten="bindings[].members" \
     --filter="bindings.members:*airbyte.io AND bindings.role:roles/bigquery.*" \
     --format="table(bindings.role, bindings.members)"
   ```

8. Prod DB-backed pages need post-apply Prod DB grants before they can work. The
   grants are not a first-apply blocker because the runtime service account does
   not exist until Pulumi creates it. They are also intentionally not managed by
   this stack because the applying identity for `internal-airbyte-ai-infra` should
   not require IAM mutation rights in the Prod DB project.

9. The MCP Artifact Registry repositories exist before pushing bootstrap
   images. If a repo already exists, leave it in place and skip its command:

   ```bash
   gcloud artifacts repositories create ops-mcp \
     --project="$PROJECT" \
     --location="$REGION" \
     --repository-format=docker \
     --description="Container images for the hosted Ops MCP server"

   gcloud artifacts repositories create cloud-mcp \
     --project="$PROJECT" \
     --location="$REGION" \
     --repository-format=docker \
     --description="Container images for the hosted Cloud MCP server"
   ```

10. MCP bootstrap images exist before Cloud Run is created:

    ```bash
    # Ops MCP
    gcloud builds submit "$REPO_ROOT" \
      --project="$PROJECT" \
      --config="$REPO_ROOT/infra/ops-webapp/cloudbuild.bootstrap-mcp.yaml"
    ```

    The image tag is defined inside `cloudbuild.bootstrap-mcp.yaml` as
    `us-west3-docker.pkg.dev/internal-airbyte-ai-infra/ops-mcp/ops-mcp:latest`.

    Cloud MCP uses a separate source repo (PyAirbyte). Its bootstrap image
    must be built and pushed to
    `us-west3-docker.pkg.dev/internal-airbyte-ai-infra/cloud-mcp/cloud-mcp:latest`
    before first apply. See the PyAirbyte repo for build instructions.

11. The Ops MCP OAuth client secret container, runtime access grant, and at
    least one secret version exist before Cloud Run is created. Create the
    Keycloak client first (see
    [`docs/keycloak-client-admin.md`](../../docs/keycloak-client-admin.md)),
    then bootstrap the secret:

    ```bash
    gcloud secrets describe ops-mcp-oauth-client-secret --project="$PROJECT" >/dev/null 2>&1 || \
      gcloud secrets create ops-mcp-oauth-client-secret \
        --project="$PROJECT" \
        --replication-policy=automatic

    gcloud secrets add-iam-policy-binding ops-mcp-oauth-client-secret \
      --project="$PROJECT" \
      --member="serviceAccount:${RUNTIME_SA}" \
      --role="roles/secretmanager.secretAccessor"

    gcloud secrets versions add ops-mcp-oauth-client-secret \
      --project="$PROJECT" \
      --data-file="$OPS_MCP_OAUTH_SECRET_FILE"
    ```

12. The Cloud MCP OAuth client secret container, runtime access grant, and at
    least one secret version exist before Cloud Run is created. Create the
    Keycloak client first (see
    [`docs/keycloak-client-admin.md`](../../docs/keycloak-client-admin.md)),
    then bootstrap the secret:

    ```bash
    gcloud secrets describe cloud-mcp-oauth-client-secret --project="$PROJECT" >/dev/null 2>&1 || \
      gcloud secrets create cloud-mcp-oauth-client-secret \
        --project="$PROJECT" \
        --replication-policy=automatic

    gcloud secrets add-iam-policy-binding cloud-mcp-oauth-client-secret \
      --project="$PROJECT" \
      --member="serviceAccount:${RUNTIME_SA}" \
      --role="roles/secretmanager.secretAccessor"

    gcloud secrets versions add cloud-mcp-oauth-client-secret \
      --project="$PROJECT" \
      --data-file="$CLOUD_MCP_OAUTH_SECRET_FILE"
    ```

## Stage 1: provision infrastructure

After Stage 0 is complete and this PR is merged, run the repository's Pulumi
apply workflow for the `ops-webapp` stack. Normal bootstrap does not require a
local Pulumi apply.

## Stage 2: grant Prod DB access after apply

After Pulumi creates `ops-webapp-sa`, ask a Prod DB project administrator to run
these grants:

```bash
export RUNTIME_SA="ops-webapp-sa@internal-airbyte-ai-infra.iam.gserviceaccount.com"

gcloud projects add-iam-policy-binding prod-ab-cloud-proj \
  --member="serviceAccount:${RUNTIME_SA}" \
  --role="roles/cloudsql.client" \
  --condition=None

gcloud secrets add-iam-policy-binding CONNECTION_RETRIEVER_PG_CONNECTION_DETAILS \
  --project=587336813068 \
  --member="serviceAccount:${RUNTIME_SA}" \
  --role="roles/secretmanager.secretAccessor"
```

These commands grant the Ops Webapp runtime identity access to the Prod DB Cloud
SQL control plane and the existing connection-details secret. They do not change
the Cloud SQL instance, public/private IP settings, VPC peering, firewall rules,
or Prod DB network configuration.

The deployed app uses the Cloud SQL Python Connector against the replica's public
IP. This avoids adding a private network path from `internal-airbyte-ai-infra` to
`projects/prod-ab-cloud-proj/global/networks/prod-main-vpc`. Cloud SQL Proxy and
Tailscale are not required for Prod DB query tools. See
<https://docs.cloud.google.com/sql/docs/postgres/language-connectors>.

## Deployment workflow handoff

After Pulumi creates the services, the deploy command workflows should only
build, push, and roll out new images. Infrastructure creation belongs in this
Pulumi project, not the deploy workflows.

- **Ops Webapp**: `deploy-ops-webapp-command.yml` builds and deploys `ops-webapp`.
- **Ops MCP**: `deploy-mcp-command.yml` builds and deploys `ops-mcp` (or
  `ops-mcp-preview` for slash-command previews). This workflow also supports
  `cloud-mcp` via the `mcp-server` input parameter.
