Deployment, Operations, and Observability

Overview

This deployment slice covers how the application is built into production images, pushed to Artifact Registry, and rolled out onto the Compute Engine host with Docker Compose. It also covers the schema-apply step that must run before production form submissions depend on new tables, and the edge routing split that sends browser traffic through Fastly and Google Cloud while separating frontend and API paths.

Operationally, the production stack is configured so static frontend assets can be cached at the edge, API paths can stay fresh, and failed backend writes can fall back to local persistence when the Next.js application route is used. Observability is bootstrapped in the frontend through OpenTelemetry and conditional Sentry imports, with runtime telemetry controlled by environment variables in frontend/.env.local.example.

Architecture Overview

flowchart LR
  subgraph CICD[CI CD Delivery]
    GitHubActions[GitHub Actions]
    ArtifactRegistry[Artifact Registry]
  end

  subgraph Edge[Edge Routing]
    Fastly[Fastly]
    LoadBalancer[Google Cloud HTTPS Load Balancer]
  end

  subgraph Runtime[Production Runtime]
    Frontend[Frontend Container]
    Backend[Backend Container]
    MySQL[MySQL Container]
    Compose[Docker Compose Prod Stack]
    SchemaApply[platform database apply schema]
  end

  subgraph Observability[Observability]
    Instrumentation[frontend instrumentation ts]
    OTel[OpenTelemetry]
    Sentry[Sentry]
  end

  Browser[Browser] --> Fastly
  Fastly --> LoadBalancer
  LoadBalancer --> Frontend
  LoadBalancer --> Backend

  GitHubActions --> ArtifactRegistry
  ArtifactRegistry --> Compose
  Compose --> Frontend
  Compose --> Backend
  Compose --> MySQL

  SchemaApply --> MySQL

  Frontend --> Instrumentation
  Instrumentation --> OTel
  Instrumentation --> Sentry

Production Image Builds

documents/deployment.md, frontend/Dockerfile, frontend/Dockerfile.local

The production deployment uses GitHub Actions to build and push container images to Artifact Registry. The backend and frontend images are tagged with latest under us-central1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/mvp-repo/..., then pulled on the Compute Engine host during rollout.

File Purpose Build or runtime behavior
frontend/Dockerfile Production frontend image Builds with node:20-alpine, accepts NEXT_PUBLIC_BACKEND_API_URL and NEXT_PUBLIC_AMPLITUDE_API_KEY as build arguments, runs npm run build, then starts with npm run start on port 3000.
frontend/Dockerfile.local Local frontend container Uses node:20-alpine, installs dependencies, and starts the dev server with npm run dev on port 3000.
documents/deployment.md Deployment runbook Defines the GitHub Actions build-and-push workflow and the Compute Engine rollout sequence.

Artifact Registry image tags

[!NOTE] documents/deployment.md describes a .env.production file with VITE_BACKEND_API_URL and ALLOWED_ORIGINS, while frontend/Dockerfile consumes NEXT_PUBLIC_BACKEND_API_URL and NEXT_PUBLIC_AMPLITUDE_API_KEY at build time. The production image therefore depends on the NEXT_PUBLIC_... build arguments, not the VITE_... names in the deployment note.

The documented image targets are:

  • us-central1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/mvp-repo/backend:latest
  • us-central1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/mvp-repo/frontend:latest

Those tags are pulled on the Compute Engine host after gcloud auth configure-docker us-central1-docker.pkg.dev has been run on that machine.

Production Rollout on Compute Engine

documents/deployment.md, platform/deployment/prod/ docker compose files

The production rollout is SSH-driven and runs on the Compute Engine instance. The documented sequence is:

  1. SSH into the instance.
  2. Pull the latest Git branch state with git fetch origin main && git rebase origin/main.
  3. Stop the current stack with docker-compose down.
  4. Authenticate Docker to Artifact Registry with gcloud auth configure-docker us-central1-docker.pkg.dev.
  5. Pull the fresh images with docker-compose pull.
  6. Restart the stack with docker-compose up -d --build.

Rollout sequence

sequenceDiagram
  participant GA as GitHub Actions
  participant AR as Artifact Registry
  participant CE as Compute Engine
  participant DC as Docker Compose
  participant DB as MySQL

  GA->>AR: Push backend and frontend images
  CE->>CE: git fetch origin main and git rebase origin main
  CE->>DC: docker compose down
  CE->>CE: gcloud auth configure docker us central1 docker pkg dev
  CE->>DC: docker compose pull
  CE->>DC: docker compose up -d --build
  DC->>DB: Start or restart database container
  DC-->>CE: Services ready

Deployment secrets

The runbook lists these repository secrets for CI/CD:

Secret Purpose
GCP_SA_KEY Service account key used by GitHub Actions
GCP_PROJECT_ID Artifact Registry and GCP project selection
COMPUTE_ENGINE_IP SSH target for rollout
COMPUTE_ENGINE_USER SSH username for the Compute Engine host
SSH_PRIVATE_KEY Private key used by appleboy/ssh-action

Schema Rollout

documents/deployment.md, platform/database/apply_schema.sh

Production database initialization only runs when the persistent MySQL volume is created for the first time. For long-lived databases, schema changes must be applied explicitly after the code has been pulled onto the Compute Engine instance.

The documented helper is:

cd ~/MVP_fastapi
MYSQL_USER="myuser" \
MYSQL_PASSWORD="mypassword" \
platform/database/apply_schema.sh

The helper reuses the Docker Compose setup to execute platform/database/1_create_database.sql inside the running MySQL container. It is explicitly described as safe to run multiple times, and it creates missing tables such as phone_applications and applications.

Schema rollout verification

The runbook verifies the schema with a direct MySQL check inside the container:

docker-compose -f platform/deployment/prod/docker-compose.yml exec mysql \
  bash -c "MYSQL_PWD=\"$MYSQL_PASSWORD\" mysql -u\"$MYSQL_USER\" -e 'USE platform; SHOW TABLES LIKE \"%applications\";'"

Schema rollout order

  1. Pull the latest code on the Compute Engine host.
  2. Run platform/database/apply_schema.sh.
  3. Verify the expected tables exist.
  4. Restart or continue the production stack.
  5. Only then rely on application pages that submit to /api/v1/applications.

Schema rollout flow

sequenceDiagram
  participant Ops as Operator
  participant Host as Compute Engine Host
  participant Compose as Docker Compose
  participant MySQL as MySQL Container
  participant App as Application Submission

  Ops->>Host: Pull latest code
  Ops->>Host: Run platform database apply schema
  Host->>Compose: Reuse compose config
  Compose->>MySQL: Execute 1_create_database sql
  MySQL-->>Compose: Create missing tables
  Ops->>MySQL: Verify applications tables
  App->>Compose: Submit form data
  Compose-->>App: Persist succeeds after schema exists

Traffic Routing and Request Flow

[!IMPORTANT] The deployment notes tie this step directly to the /application submission flow. If the table does not exist, backend inserts fail and the production form cannot persist records correctly.

documents/deployment.md, frontend/app/api/applications/route.ts, frontend/app/api/v1/applications/route.ts

The production traffic model is split across Fastly and Google Cloud:

  • Fastly fronts app.joinlita.com.
  • The Google Cloud HTTPS Load Balancer is the origin.
  • TLS certificates are managed by Fastly.
  • A DNS CNAME, ALIAS, or ANAME maps app.joinlita.com to the Fastly service domain.
  • Static frontend assets are cached at the edge.
  • API paths such as /api/* bypass cache.

The load balancer then splits traffic by host and path rule:

Rule Target
/api/* mvp-backend-api
all other requests mvp-backend

The documented backend services use named ports:

Backend service Purpose Named port Port
mvp-backend Frontend UI http-frontend 80
mvp-backend-api API endpoints http-backend 8000

Request routing sequence

sequenceDiagram
  participant Browser as Browser
  participant Fastly as Fastly
  participant LB as Google Cloud HTTPS Load Balancer
  participant Frontend as Frontend Service
  participant API as API Service

  Browser->>Fastly: Request app joinlita dot com
  Fastly->>LB: Forward origin request
  alt Static or app page
    LB->>Frontend: Route non api path
    Frontend-->>LB: HTML or static asset
  else API request
    LB->>API: Route api path
    API-->>LB: JSON response
  end
  LB-->>Fastly: Response
  Fastly-->>Browser: Cached or uncached response

Fastly caching model

Path type Cache behavior
Static frontend assets Cached
/api/* Bypassed

Runtime Configuration and Observability

[!NOTE] documents/deployment.md sends /api/* to the Google Cloud API backend service, while the Next.js frontend also defines frontend/app/api/v1/applications/route.ts. The external routing split therefore places public /api traffic on the Google Cloud API origin, while the frontend route remains part of the Next.js application container and its local or alternate deployment paths.

frontend/.env.local.example, frontend/instrumentation.ts, documents/deployment.md

The frontend runtime is configured through environment variables for authentication, telemetry, and build-time behavior. The repository includes a detailed example file for local and environment setup, while the deployment notes define additional production-specific values for the application storage fallback path.

Frontend runtime variables

Variable Purpose
AUTH0_DOMAIN Auth0 tenant domain without protocol
AUTH0_CLIENT_ID Public Auth0 client identifier
AUTH0_CLIENT_SECRET Auth0 client secret
AUTH0_ORGANIZATION_ID Optional Auth0 organization identifier
NEXT_PUBLIC_AUTH0_ORGANIZATION_ID Client-side fallback for organization-aware login redirects
AUTH0_SECRET Cookie encryption secret
AUTH0_BASE_URL Local or deployed base URL
AUTH0_ISSUER_BASE_URL Auth0 issuer base URL
AUTH0_MANAGEMENT_CLIENT_ID Auth0 Management API client ID
AUTH0_MANAGEMENT_CLIENT_SECRET Auth0 Management API client secret
AUTH0_MANAGEMENT_AUDIENCE Auth0 Management API audience
NEXT_PUBLIC_SENTRY_DSN Client-side Sentry DSN
SENTRY_DSN Server-side Sentry DSN
NEXT_PUBLIC_SENTRY_ENVIRONMENT Client-side Sentry environment
SENTRY_ENVIRONMENT Server-side Sentry environment
NEXT_PUBLIC_SENTRY_TRACES_SAMPLE_RATE Client-side tracing sample rate
SENTRY_TRACES_SAMPLE_RATE Server-side tracing sample rate
NEXT_PUBLIC_SENTRY_REPLAYS_SESSION_SAMPLE_RATE Session replay sample rate
NEXT_PUBLIC_SENTRY_REPLAYS_ON_ERROR_SAMPLE_RATE Replay-on-error sample rate
SENTRY_AUTH_TOKEN Sentry auth token
SENTRY_ORG Sentry organization slug
SENTRY_PROJECT Sentry project slug
NEXT_PUBLIC_AMPLITUDE_API_KEY Frontend analytics key
OTEL_SERVICE_NAME OpenTelemetry service name
OTEL_EXPORTER_OTLP_ENDPOINT OpenTelemetry exporter endpoint
OTEL_EXPORTER_OTLP_HEADERS OpenTelemetry exporter headers

Production storage variables

The deployment notes also define submission storage overrides for the application route:

Variable Purpose
APPLICATIONS_DATA_GCS_BUCKET Google Cloud Storage bucket for persisted application submissions
APPLICATIONS_DATA_GCS_OBJECT Optional object name, defaulting to applications.json

When those variables are omitted, the route falls back to the local data/applications.json file for development convenience.

OpenTelemetry and Sentry bootstrap

frontend/instrumentation.ts initializes telemetry during frontend startup.

Property Type Description
serviceName string Set from OTEL_SERVICE_NAME or defaults to lita-frontend
Method Description
register Registers OpenTelemetry with registerOTel, then conditionally imports ./sentry.server.config for Node.js runtime or ./sentry.edge.config for Edge runtime.

Telemetry startup sequence

sequenceDiagram
  participant Next as Next js runtime
  participant Inst as instrumentation ts
  participant OTel as OpenTelemetry
  participant Sentry as Sentry config

  Next->>Inst: register
  Inst->>OTel: registerOTel with serviceName
  alt nodejs runtime
    Inst->>Sentry: import sentry server config
  else edge runtime
    Inst->>Sentry: import sentry edge config
  end

Deployment API Surface

GET /api/v1/applications

frontend/app/api/v1/applications/route.ts, frontend/app/api/applications/route.ts

This endpoint serves application records as JSON by default and can also render an HTML table when format=html is provided. It reads backend data when available, falls back to stored application records, and returns an empty array if nothing exists.

List Applications

{
    "title": "List Applications",
    "description": "Returns application records as JSON for deployment checks and dashboard data loading. The same handler can also render HTML when format=html is requested.",
    "method": "GET",
    "baseUrl": "<AppBaseUrl>",
    "endpoint": "/api/v1/applications",
    "headers": [],
    "queryParams": [
        {
            "key": "format",
            "value": "json",
            "required": false
        }
    ],
    "pathParams": [],
    "bodyType": "none",
    "requestBody": "",
    "formData": [],
    "rawBody": "",
    "responses": {
        "200": {
            "description": "Success",
            "body": "[\n    {\n        \"id\": 101,\n        \"identifier\": \"app-101\",\n        \"email\": \"[email protected]\",\n        \"phone\": \"+1 555 0100\",\n        \"country\": \"US\",\n        \"employment_status\": \"Employed\",\n        \"financing_consent\": \"granted\",\n        \"full_name\": \"Demo Applicant\",\n        \"date_of_birth\": \"1990-04-12\",\n        \"profession\": \"Product Manager\",\n        \"current_employer\": \"Lita Housing\",\n        \"job_title\": \"Operations Lead\",\n        \"industry\": \"Real Estate\",\n        \"website\": \"https://example.com\",\n        \"linkedin\": \"https://linkedin.com/in/demo-applicant\",\n        \"agree_policy\": true,\n        \"receive_updates\": false,\n        \"skipped_connect_accounts\": false,\n        \"connected_accounts\": {\n            \"personal\": [],\n            \"business\": [],\n            \"investments\": []\n        },\n        \"metadata\": {\n            \"application_reference\": \"AP_1A2B3C4\",\n            \"source\": \"production\"\n        },\n        \"created_at\": \"2026-03-31T12:00:00.000Z\"\n    }\n]"
        }
    }
}

POST /api/v1/applications

frontend/app/api/v1/applications/route.ts, frontend/app/api/applications/route.ts

This endpoint validates the incoming payload, proxies it to the backend when the backend is reachable, and falls back to local JSON persistence when the backend write fails. In the fallback path it assigns a numeric id, stamps created_at, writes the record to the local storage file, and returns a synthesized 201 response.

Create Application

{
    "title": "Create Application",
    "description": "Submits an application payload to the backend and falls back to local persistence when backend submission fails.",
    "method": "POST",
    "baseUrl": "<AppBaseUrl>",
    "endpoint": "/api/v1/applications",
    "headers": [
        {
            "key": "Content-Type",
            "value": "application/json",
            "required": true
        }
    ],
    "queryParams": [],
    "pathParams": [],
    "bodyType": "json",
    "requestBody": "{\n    \"id\": 101,\n    \"email\": \"[email protected]\",\n    \"phone\": \"+1 555 0100\",\n    \"country\": \"US\",\n    \"employment_status\": \"Employed\",\n    \"financing_consent\": \"granted\",\n    \"full_name\": \"Demo Applicant\",\n    \"date_of_birth\": \"1990-04-12\",\n    \"profession\": \"Product Manager\",\n    \"current_employer\": \"Lita Housing\",\n    \"job_title\": \"Operations Lead\",\n    \"industry\": \"Real Estate\",\n    \"website\": \"https://example.com\",\n    \"linkedin\": \"https://linkedin.com/in/demo-applicant\",\n    \"agree_policy\": true,\n    \"receive_updates\": false,\n    \"skipped_connect_accounts\": false,\n    \"connected_accounts\": {\n        \"personal\": [\n            {\n                \"id\": \"acc_001\",\n                \"name\": \"Primary Checking\",\n                \"type\": \"checking\"\n            }\n        ],\n        \"business\": [],\n        \"investments\": []\n    },\n    \"metadata\": {\n        \"application_reference\": \"AP_1A2B3C4\",\n        \"source\": \"production\"\n    },\n    \"created_at\": \"2026-03-31T12:05:00.000Z\"\n}",
    "formData": [],
    "rawBody": "",
    "responses": {
        "201": {
            "description": "Local fallback success",
            "body": "{\n    \"id\": 102,\n    \"created_at\": \"2026-03-31T12:05:00.000Z\",\n    \"persisted\": \"local_fallback\",\n    \"detail\": \"Backend persistence is currently unavailable. Saved locally so submission can continue.\"\n}"
        }
    }
}

Application submission flow

sequenceDiagram
  participant User as User
  participant UI as Review Page
  participant Route as api applications route
  participant Backend as Backend API
  participant Store as Local submissions file

  User->>UI: Submit application
  UI->>Route: POST application payload
  Route->>Route: Validate payload and resolve backend URLs
  Route->>Backend: Forward JSON to backend
  alt Backend accepts write
    Backend-->>Route: Created record
    Route-->>UI: 201 response from backend
  else Backend rejects or is unavailable
    Route->>Store: Read and append local submission
    Store-->>Route: Persisted locally
    Route-->>UI: 201 local fallback response
  end

Error Handling

The deployment and application-route code uses explicit fallback paths instead of failing hard when the backend is unavailable.

Location Behavior
frontend/app/api/applications/route.ts Logs a warning when backend submission fails, then stores the payload locally and returns a 201 fallback response.
frontend/app/api/applications/route.ts Returns 400 for invalid JSON payloads or validation failures.
frontend/instrumentation.ts Chooses runtime-specific telemetry imports based on NEXT_RUNTIME.
platform/database/apply_schema.sh Can be rerun safely to apply missing tables without special casing first-run behavior.

Caching Strategy

Layer Cache behavior
Fastly Caches static frontend assets and bypasses /api/*.
Next.js backend fetches Use cache: "no-store" for application submission and read paths that must stay fresh.
Schema rollout No cache; changes are applied directly into the running MySQL container.

The deployment notes also emphasize that the application data route should remain fresh on production deployments so dashboard reads reflect the latest submissions. The edge cache is therefore limited to static frontend content.

Operational Verification

The documented post-deployment checks are:

  • Open https://app.joinlita.com.
  • Confirm client-side routing such as /login and /register.
  • Verify /api/... requests route correctly through the split origin model.
  • Check Load Balancer health checks in the GCP Console.
  • Inspect backend and frontend container logs on the Compute Engine host.

Log inspection

The runbook uses gcloud compute ssh to reach the host and then checks container logs through Docker Compose. The documented command style is:

gcloud compute ssh instance-20250206-052936
cd MVP_fastapi
docker-compose -f platform/deployment/prod/docker-compose.yaml logs backend
docker-compose -f platform/deployment/prod/docker-compose.yml logs frontend

Dependencies

Dependency Role in deployment or observability
GitHub Actions Builds and pushes container images
Artifact Registry Stores versioned backend and frontend images
Compute Engine Hosts the production Docker Compose stack
Docker Compose Starts, stops, and updates containers
Fastly Edge delivery, TLS, and static asset caching
Google Cloud HTTPS Load Balancer Origin routing for frontend and API paths
MySQL container Production relational data store
Google Cloud Storage Optional storage backend for application submissions
OpenTelemetry Frontend instrumentation bootstrap
Sentry Node and Edge runtime error reporting
Auth0 Frontend authentication runtime configuration
Amplitude Frontend analytics key configuration

Testing Considerations

Scenario What to verify
Fresh deployment Images pull from Artifact Registry and containers start cleanly.
Schema update platform/database/apply_schema.sh creates missing tables before submissions are posted.
Application submission POST /api/v1/applications succeeds, or falls back to local persistence when backend writes fail.
Edge routing /api/* reaches the API backend service, while non-API traffic reaches the frontend service.
Static caching Frontend assets are cached by Fastly, but API traffic bypasses cache.
Observability startup frontend/instrumentation.ts registers OpenTelemetry and loads the correct Sentry config for the runtime.

Key Classes Reference

Class Responsibility
documents/deployment.md Production deployment runbook, routing model, and verification steps
platform/database/apply_schema.sh Applies the current schema to a long-lived production MySQL volume
frontend/Dockerfile Builds the production frontend image for Artifact Registry
frontend/Dockerfile.local Builds the local development frontend container
frontend/.env.local.example Documents runtime environment variables for Auth0, Sentry, Amplitude, and OpenTelemetry
frontend/instrumentation.ts Registers OpenTelemetry and loads runtime-specific Sentry configuration
frontend/app/api/applications/route.ts Implements application submission proxying and local fallback persistence
frontend/app/api/v1/applications/route.ts Exposes the production-facing v1 alias for the application route