Deployment, Operations, and Observability

Overview

This section covers the release workflow for the MVP stack, from pre-launch validation to post-deploy verification and rollback-oriented checks. The documented process combines backend unit tests, local smoke scripts, and Docker Compose rollouts on a Compute Engine host, with Fastly and the Google Cloud HTTPS Load Balancer handling production traffic routing.

Operationally, the goal is to catch stability regressions before launch, validate that the frontend can still reach backend APIs after deployment, and give operators a repeatable way to inspect health and container logs on the host. The workflow also includes a CPU guard and short load burst during pre-launch validation so the release can fail fast on runaway behavior before it reaches production traffic.

Architecture Overview

flowchart TD
    subgraph GitHubActions [GitHub Actions Workflow]
        GA1[DWILAR Pre Launch Stability]
        GA2[CPU guard]
        GA3[Short load burst]
    end

    subgraph LocalValidation [Local Prelaunch Validation]
        LS1[scripts lita_smoke_test.sh]
        LS2[backend pytest]
        LS3[uvicorn SQLite fallback]
        FS1[scripts prelaunch_frontend_smoke_test.sh]
        FS2[Next js app]
        FS3[Frontend health route]
    end

    subgraph ProdHost [Compute Engine Host]
        PH1[docker compose pull]
        PH2[docker compose up]
        PH3[backend container]
        PH4[frontend container]
        PH5[container logs]
    end

    GA1 --> LS1
    GA1 --> GA2
    GA1 --> GA3
    LS1 --> LS2
    LS2 --> LS3
    LS3 --> FS1
    FS1 --> FS2
    FS2 --> FS3
    GA1 --> PH1
    PH1 --> PH2
    PH2 --> PH3
    PH2 --> PH4
    PH3 --> PH5
    PH4 --> PH5

Pre-launch Stability Workflow

The documented pre-launch workflow is the DWILAR | Pre-Launch Stability GitHub Actions job. It runs the backend smoke path first, then exercises the frontend startup and health check path. The workflow is designed to run against a local FastAPI instance with the SQLite fallback, which keeps the validation loop fast and isolated from production databases.

The pre-launch job also includes a short load burst and CPU guard. The documented local inputs are LITA_CPU_LIMIT=90 and LITA_LOAD_REQUESTS=40, which let operators tune the stress check before release.

Run the workflow

  1. Open GitHub Actions → DWILAR | Pre-Launch Stability.
  2. Click Run workflow on the target branch.
  3. Optionally set the backend_url input to the environment you want to validate.

Local validation commands

Run the same smoke steps locally when you need a quick pre-flight check from the repo root:

python -m pip install -r backend/requirements.txt

NEXT_PUBLIC_BACKEND_API_URL="https://app.joinlita.com" \
LITA_CPU_LIMIT=90 LITA_LOAD_REQUESTS=40 bash scripts/lita_smoke_test.sh

NEXT_PUBLIC_BACKEND_API_URL="https://app.joinlita.com" \
BACKEND_API_URL="https://app.joinlita.com" \
bash scripts/prelaunch_frontend_smoke_test.sh

Pre-launch sequence

sequenceDiagram
    participant O as Operator
    participant G as GitHub Actions
    participant S as Smoke Script
    participant P as Pytest
    participant A as FastAPI App
    participant F as Frontend Smoke Script
    participant N as Next js App
    participant B as Backend API

    O->>G: Run DWILAR Pre Launch Stability
    G->>S: Start lita_smoke_test.sh
    S->>P: Run backend tests
    P-->>S: Test results
    S->>A: Start FastAPI with SQLite fallback
    S->>A: Create consent and token
    S->>A: Save and read snapshot
    S->>A: Score twice for idempotency
    A-->>S: Smoke responses
    G->>F: Start prelaunch_frontend_smoke_test.sh
    F->>N: Start Next js app
    F->>B: Verify frontend health route can reach backend
    B-->>F: Health response
    F-->>G: Frontend smoke result
    G-->>O: Pass or fail

Smoke Test Scripts

scripts/lita_smoke_test.sh

[!IMPORTANT] The backend smoke path is not just a startup check. It runs the targeted pytest module, starts FastAPI against the SQLite fallback, then exercises consent, token, snapshot, and scoring behavior before the frontend smoke path runs.

This script is the primary backend smoke harness referenced by the deployment documentation and Lita testing guide. It validates the backend test path first, then runs the application flow against a local FastAPI process using the SQLite fallback.

Item Details
Purpose Validate backend readiness before launch
Test command Targeted pytest module for the Lita endpoints
Runtime Local FastAPI instance with SQLite fallback
Smoke actions Create consent, issue token, save snapshot, read snapshot, score twice
Idempotency check Repeats the scoring request with the same request_id
Dependency bootstrap Automatically installs backend requirements unless disabled

Key inputs and controls:

Variable Used For Meaning
LITA_BOOTSTRAP_DEPS Dependency bootstrap Set to 0 to disable automatic pip install
LITA_CPU_LIMIT Load guard CPU ceiling used during the pre-launch load burst
LITA_LOAD_REQUESTS Load guard Number of requests used for the short stress run
NEXT_PUBLIC_BACKEND_API_URL API target Backend URL used by the smoke path

The script is documented as a single-command verification path:

./scripts/lita_smoke_test.sh

scripts/prelaunch_frontend_smoke_test.sh

This script starts the frontend and verifies that the frontend health route can reach the backend. It is the frontend counterpart to the backend smoke harness and is explicitly referenced in the deployment notes.

Item Details
Purpose Verify frontend startup and backend reachability
Runtime Starts the Next.js app
Health check Confirms the frontend health route can reach the backend
Inputs Backend URL environment variables

Key inputs:

Variable Used For Meaning
NEXT_PUBLIC_BACKEND_API_URL Frontend runtime Backend URL exposed to the frontend
BACKEND_API_URL Smoke script Backend URL used by the verification script
backend_url GitHub Actions workflow Optional target environment for the pre-launch run

Backend Validation with Pytest

The deployment guide explicitly calls for backend unit tests before deployment. The pre-launch smoke script also runs a targeted backend test module.

Documented commands

pytest tests/ -v
pytest backend/tests/interface/test_lita_endpoints.py -q

The first command is the deployment-level validation step. The second command is the targeted backend regression check used by the smoke workflow for consent, token, snapshot, and scoring coverage.

Validation scope

  • Ensures the backend test suite passes before image rollout.
  • Verifies the Lita API smoke flow against a local runtime.
  • Confirms scoring idempotency by repeating the same request_id.

Rollback-Oriented Checklist

The deployment instructions already define the canonical container restart sequence on the Compute Engine host. Use the same sequence as the base rollback path when a release needs to be backed out or rechecked.

Checklist

  1. SSH into the Compute Engine host:
   gcloud compute ssh instance-20250206-052936
  1. Change into the deployment directory:
   cd ~/MVP_fastapi/platform/deployment/prod
  1. Stop the current stack:
   docker-compose down
  1. Re-authenticate Docker to Artifact Registry if needed:
   gcloud auth configure-docker us-central1-docker.pkg.dev
  1. Pull the images for the checked-out revision:
   docker-compose pull
  1. Bring the stack back up:
   docker-compose up -d --build
  1. Check the frontend health route:
   curl http://localhost:3000/api/ready
  1. Confirm backend connectivity through the frontend proxy routes using a known application identifier from the deployed dataset.

  2. Inspect logs if any route check fails.

Post-Deploy Verification and Observability

Post-deploy checks are a combination of browser access, API routing validation, load balancer health, and container log inspection.

What to verify

  • The application is reachable at https://app.joinlita.com.
  • Client-side routes such as /login and /register still render correctly.
  • Requests under https://app.joinlita.com/api/... are routed correctly.
  • GCP Load Balancer health checks show healthy backend status.
  • Backend and frontend container logs are clean enough to rule out startup or routing failures.

Container log inspection on the Compute Engine host

The deployment guide shows host-side log inspection with Docker Compose:

gcloud compute ssh instance-20250206-052936
cd MVP_fastapi
docker-compose -f platform/deployment/prod/docker-compose.yaml logs backend
docker-compose -f platform/deployment/prod/docker-compose.yml logs frontend

Observability checklist

[!NOTE] The documented log commands mix docker-compose.yaml and docker-compose.yml. The deploy step itself uses the docker-compose.yml path, so copy the log command carefully and keep the same compose file name that is present on the host.

  • Check Google Cloud Load Balancer health in the console.
  • Inspect backend container logs for API or startup failures.
  • Inspect frontend container logs for route or backend proxy failures.
  • Re-run the frontend health route check after any container restart.
  • If a verification fails, repeat the pull/up sequence after correcting the checked-out revision.

Operational API Checks

These routes are useful as post-deploy probes because they exercise the frontend-to-backend path and surface backend reachability problems.

Get Applicant Account Coverage

Get Applicant Account Coverage

{
    "title": "Get Applicant Account Coverage",
    "description": "Checks whether submitted connected accounts cover the accounts detected by Plaid and proxies the backend response through the frontend",
    "method": "GET",
    "baseUrl": "<FrontendBaseUrl>",
    "endpoint": "/api/v1/applicants/{id}/account-coverage",
    "headers": [],
    "queryParams": [],
    "pathParams": [
        {
            "key": "id",
            "type": "string",
            "required": true,
            "description": "Applicant identifier"
        }
    ],
    "bodyType": "none",
    "requestBody": "",
    "formData": [],
    "rawBody": "",
    "responses": {
        "200": {
            "description": "Success",
            "body": "{\n    \"application_id\": 123,\n    \"status\": \"incomplete\",\n    \"reported_accounts\": 2,\n    \"detected_accounts\": 3,\n    \"missing_accounts\": 1,\n    \"coverage_ratio\": 0.67,\n    \"details\": [\n        {\n            \"connection_name\": \"Example Bank\",\n            \"institution_name\": \"Example Bank\",\n            \"reported_accounts\": 2,\n            \"detected_accounts\": 3,\n            \"missing_accounts\": 1,\n            \"status\": \"incomplete\"\n        }\n    ],\n    \"errors\": []\n}"
        },
        "400": {
            "description": "Invalid applicant id",
            "body": "{\n    \"error\": \"Invalid applicant id.\"\n}"
        },
        "500": {
            "description": "Backend API URL not configured",
            "body": "{\n    \"error\": \"Backend API URL is not configured. Set NEXT_PUBLIC_BACKEND_API_URL or BACKEND_API_URL.\"\n}"
        },
        "502": {
            "description": "Backend unreachable",
            "body": "{\n    \"error\": \"Failed to reach backend API.\"\n}"
        }
    }
}

Get Applicant Transactions

Get Applicant Transactions

{
    "title": "Get Applicant Transactions",
    "description": "Returns transaction history for a selected applicant after combining backend, remote, and locally stored application data",
    "method": "GET",
    "baseUrl": "<FrontendBaseUrl>",
    "endpoint": "/api/v1/applicants/{id}/transactions",
    "headers": [],
    "queryParams": [
        {
            "key": "start_date",
            "type": "string",
            "required": false,
            "description": "Lower bound for the transaction date filter"
        },
        {
            "key": "end_date",
            "type": "string",
            "required": false,
            "description": "Upper bound for the transaction date filter"
        }
    ],
    "pathParams": [
        {
            "key": "id",
            "type": "string",
            "required": true,
            "description": "Applicant identifier"
        }
    ],
    "bodyType": "none",
    "requestBody": "",
    "formData": [],
    "rawBody": "",
    "responses": {
        "200": {
            "description": "Success",
            "body": "{\n    \"applicant_id\": \"123\",\n    \"start_date\": null,\n    \"end_date\": null,\n    \"transactions\": [\n        {\n            \"amount\": -42.5,\n            \"date\": \"2024-05-01\",\n            \"currency_code\": \"USD\",\n            \"status\": \"posted\",\n            \"account_name\": \"Checking\",\n            \"connection_name\": \"Example Bank\",\n            \"description\": \"Coffee shop\",\n            \"categories\": [\n                \"food_and_drink\",\n                \"coffee_shop\"\n            ],\n            \"provider\": \"plaid\",\n            \"metadata\": {\n                \"amount\": -42.5,\n                \"date\": \"2024-05-01\",\n                \"currency_code\": \"USD\",\n                \"status\": \"posted\",\n                \"account_name\": \"Checking\",\n                \"connection_name\": \"Example Bank\",\n                \"description\": \"Coffee shop\",\n                \"categories\": [\n                    \"food_and_drink\",\n                    \"coffee_shop\"\n                ],\n                \"provider\": \"plaid\"\n            }\n        }\n    ]\n}"
        },
        "400": {
            "description": "Invalid request",
            "body": "{\n    \"error\": \"Invalid applicant id.\"\n}"
        },
        "404": {
            "description": "Application missing or no transaction history",
            "body": "{\n    \"error\": \"No transaction history is available for this applicant yet. Connect and sync bank accounts first.\"\n}"
        },
        "502": {
            "description": "Backend unreachable",
            "body": "{\n    \"error\": \"Failed to reach backend API.\"\n}"
        }
    }
}

Dependencies

[!NOTE] The frontend proxy skips backend 404 responses while trying base URL candidates. If every candidate returns 404, the route can still end in a 502 style failure message instead of surfacing a plain not found response.

  • pytest for backend validation.
  • uvicorn for local FastAPI startup during smoke tests.
  • Docker and Docker Compose for the host rollout.
  • Google Cloud CLI for Compute Engine SSH access and Artifact Registry auth.
  • GitHub Actions for the pre-launch stability workflow.
  • Google Cloud HTTPS Load Balancer health checks and console inspection.
  • Fastly in front of the public application origin.

Testing Considerations

  • Run the backend pytest path before any image rollout.
  • Use the smoke scripts to validate both the backend flow and the frontend-to-backend path.
  • Keep LITA_CPU_LIMIT and LITA_LOAD_REQUESTS aligned with the release environment when using the CPU guard.
  • Confirm the frontend health route passes before and after any compose restart.
  • Inspect host-side logs any time a route check returns a 502 or the frontend cannot reach backend data.

Key Classes Reference

Class Responsibility
lita_smoke_test.sh Runs backend pytest, starts FastAPI on the SQLite fallback, and exercises consent, snapshot, and scoring smoke checks
prelaunch_frontend_smoke_test.sh Starts the frontend and verifies the health route can reach the backend
route.ts Proxies applicant account coverage checks through the frontend to the backend
route.ts Combines backend and stored data to return applicant transaction history
deployment.md Documents the release, post-deploy, and log inspection workflow