Deployment, Operations, and Observability

Overview

This section covers the release path for Lita’s web application from verified code to a running production stack. The documented workflow uses GitHub Actions to run the backend test gate, publish Docker images to Artifact Registry, and then trigger an SSH-based rollout onto the Compute Engine host that runs docker-compose in platform/deployment/prod.

Operationally, the deployment notes also define how to verify the rollout, where to inspect container logs, and how to run pre-launch stability checks before launch. The observability surface is centered on the frontend’s instrumentation.ts, which wires OpenTelemetry and Sentry at runtime, plus the deployment-time environment variables that control telemetry behavior.

Architecture Overview

flowchart TB
    subgraph GitHubActions[GitHub Actions CI CD]
        Tests[Backend pytest gate]
        BuildBackend[Build and push backend image]
        BuildFrontend[Build and push frontend image]
        DeploySSH[Deploy via SSH]
    end

    subgraph ArtifactRegistry[Artifact Registry]
        BackendImage[backend latest image]
        FrontendImage[frontend latest image]
    end

    subgraph ComputeEngine[Compute Engine rollout host]
        DeployDir[platform deployment prod]
        PullImages[docker compose pull]
        Recreate[docker compose up -d --build]
    end

    subgraph FrontendRuntime[Frontend runtime container]
        ProdDockerfile[frontend Dockerfile]
        LocalDockerfile[frontend Dockerfile local]
        Telemetry[instrumentation ts]
    end

    subgraph Observability[Observability sinks]
        OpenTelemetry[OpenTelemetry]
        Sentry[Sentry]
        Logs[Container logs]
    end

    Tests --> BuildBackend
    Tests --> BuildFrontend
    BuildBackend --> BackendImage
    BuildFrontend --> FrontendImage
    DeploySSH --> DeployDir
    DeployDir --> PullImages
    PullImages --> Recreate
    Recreate --> FrontendRuntime
    FrontendRuntime --> Telemetry
    Telemetry --> OpenTelemetry
    Telemetry --> Sentry
    ComputeEngine --> Logs

CI/CD Image Build and Publishing

The deployment document defines a two-image publish step: one image for the backend and one for the frontend. Both are built with docker/build-push-action@v4 and pushed to Artifact Registry under us-central1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/mvp-repo/....

GitHub Actions release gates

Stage Tooling Behavior
Test gate pytest tests/ -v Runs backend unit tests before any image publish step.
Backend image publish docker/build-push-action@v4 Builds from ./backend, pushes backend:latest.
Frontend image publish docker/build-push-action@v4 Builds from ./frontend, pushes frontend:latest.
Rollout appleboy/[email protected] Connects to the Compute Engine host and runs the deployment script.

Artifact Registry authentication surface

The documented GitHub Actions setup depends on these repository secrets:

Secret Purpose
GCP_SA_KEY Service account credential used by the workflow.
GCP_PROJECT_ID Project ID used in Artifact Registry image tags.
COMPUTE_ENGINE_IP SSH target for the rollout step.
COMPUTE_ENGINE_USER SSH username for the Compute Engine host.
SSH_PRIVATE_KEY Private key passed to appleboy/ssh-action.

The rollout script also authenticates Docker on the VM with:

gcloud auth configure-docker us-central1-docker.pkg.dev

That step is required before docker-compose pull can fetch the published images from Artifact Registry.

Publish commands documented in the deployment guide

- name: Build and Push Backend Image
  uses: docker/build-push-action@v4
  with:
    context: ./backend
    push: true
    tags: us-central1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/mvp-repo/backend:latest

- name: Build and Push Frontend Image
  uses: docker/build-push-action@v4
  with:
    context: ./frontend
    push: true
    tags: us-central1-docker.pkg.dev/${{ secrets.GCP_PROJECT_ID }}/mvp-repo/frontend:latest

Rollout Workflow Over SSH

The production deployment is executed on the Compute Engine instance after the images are pushed. The SSH action runs a small release script inside ~/MVP_fastapi/platform/deployment/prod.

Rollout sequence

sequenceDiagram
    participant Dev as Developer
    participant GH as GitHub Actions
    participant AR as Artifact Registry
    participant VM as Compute Engine host
    participant DC as docker compose

    Dev->>GH: Push to protected branch
    GH->>GH: Run pytest gate
    GH->>AR: Build and push backend image
    GH->>AR: Build and push frontend image
    GH->>VM: appleboy ssh action
    VM->>VM: git fetch origin main and git rebase origin main
    VM->>VM: docker compose down
    VM->>VM: gcloud auth configure docker us-central1 docker pkg dev
    VM->>DC: docker compose pull
    VM->>DC: docker compose up -d --build
    DC-->>VM: Containers restarted with published images
    VM-->>GH: Rollout completes

Remote commands

cd ~/MVP_fastapi/platform/deployment/prod
git fetch origin main && git rebase origin/main
docker-compose down
gcloud auth configure-docker us-central1-docker.pkg.dev
docker-compose pull
docker-compose up -d --build

Rollout notes

  • git fetch plus git rebase keeps the VM aligned with origin/main.
  • docker-compose down clears the previous stack before the new images are pulled.
  • docker-compose pull consumes the Artifact Registry images published by GitHub Actions.
  • docker-compose up -d --build brings the services back up in detached mode.

Production and Local Container Composition

The repository shows two hardened container patterns for the frontend: a production multi-stage build and a dedicated local hot-reload build. These are the clearest documented examples of how the application distinguishes release images from development images.

Frontend Production Dockerfile

frontend/Dockerfile

The production build uses a two-stage image to separate build-time dependencies from the runtime container.

Stage Base Image Behavior
builder node:20-alpine Installs dependencies, copies source, and runs npm run build.
runner node:20-alpine Copies the built output, installs production dependencies, and starts the app.

Build-time inputs

Name Type Purpose
NEXT_PUBLIC_BACKEND_API_URL ARG / ENV Injected into the build so the client bundle knows the backend base URL.
NEXT_PUBLIC_AMPLITUDE_API_KEY ARG / ENV Injected into the build for client-side analytics config.
NODE_ENV ENV Set to production in the runtime stage.

Runtime composition

File element Behavior
COPY --from=builder /app/.next ./.next Ships the compiled Next.js build output.
COPY --from=builder /app/public ./public Ships static assets.
RUN npm install --production --legacy-peer-deps Installs runtime-only dependencies.
EXPOSE 3000 Declares the application port.
CMD ["npm", "run", "start"] Starts the production server.

Public methods

None. This file defines the container image build and runtime entrypoint only.

Frontend Local Dockerfile

frontend/Dockerfile.local

This file is the local hot-reload variant for development. It keeps the source tree in the container and starts the app with npm run dev.

File element Behavior
FROM node:20-alpine Uses the same base image family as production.
RUN npm install --legacy-peer-deps Installs the full dependency set needed for local development.
COPY . . Copies the entire frontend source tree.
EXPOSE 3000 Declares the dev server port.
CMD ["npm", "run", "dev"] Starts the hot-reload development server.

Production vs local difference

Aspect frontend/Dockerfile frontend/Dockerfile.local
Build style Multi-stage Single-stage
Runtime dependencies Production only Full dev dependency set
Startup command npm run start npm run dev
Intended use Release image Local development and hot reload

Frontend build-time environment contract

frontend/.env.local.example

This file defines the frontend’s local telemetry and runtime configuration surface.

Variable Purpose
AUTH0_DOMAIN Auth0 tenant domain.
AUTH0_CLIENT_ID Auth0 client identifier.
AUTH0_CLIENT_SECRET Auth0 client secret.
AUTH0_ORGANIZATION_ID Optional organization-aware login setting.
NEXT_PUBLIC_AUTH0_ORGANIZATION_ID Client-side organization fallback.
AUTH0_SECRET Cookie encryption secret.
AUTH0_BASE_URL Local application base URL.
AUTH0_ISSUER_BASE_URL Auth0 issuer base URL.
AUTH0_MANAGEMENT_CLIENT_ID Auth0 management client ID.
AUTH0_MANAGEMENT_CLIENT_SECRET Auth0 management client secret.
AUTH0_MANAGEMENT_AUDIENCE Auth0 management API audience.
NEXT_PUBLIC_SENTRY_DSN Browser-side Sentry DSN.
SENTRY_DSN Server-side Sentry DSN.
NEXT_PUBLIC_SENTRY_ENVIRONMENT Browser-side Sentry environment label.
SENTRY_ENVIRONMENT Server-side Sentry environment label.
NEXT_PUBLIC_SENTRY_TRACES_SAMPLE_RATE Browser-side trace sampling.
SENTRY_TRACES_SAMPLE_RATE Server-side trace sampling.
NEXT_PUBLIC_SENTRY_REPLAYS_SESSION_SAMPLE_RATE Browser replay sampling.
NEXT_PUBLIC_SENTRY_REPLAYS_ON_ERROR_SAMPLE_RATE Browser replay-on-error sampling.
SENTRY_AUTH_TOKEN Sentry upload/auth token.
SENTRY_ORG Sentry organization slug.
SENTRY_PROJECT Sentry project slug.
NEXT_PUBLIC_AMPLITUDE_API_KEY Browser analytics key.
OTEL_SERVICE_NAME OpenTelemetry service name.
OTEL_EXPORTER_OTLP_ENDPOINT OTLP endpoint for telemetry export.
OTEL_EXPORTER_OTLP_HEADERS Extra OTLP export headers.

Pre-Launch Stability Checks

The deployment document defines a separate scheduled workflow named DWILAR | Pre-Launch Stability. It is designed to catch release regressions before launch by running smoke tests, a short load burst, and a CPU guard against a local FastAPI instance.

Workflow behavior

Check Script Purpose
Backend smoke scripts/lita_smoke_test.sh Exercises the backend using the SQLite fallback.
Frontend smoke scripts/prelaunch_frontend_smoke_test.sh Starts the Next.js app and verifies the frontend health route can reach the backend.
Load and CPU guard Part of the scheduled workflow Surfaces runaway CPU spikes and stability regressions before launch.

Local execution examples from the deployment guide

python -m pip install -r backend/requirements.txt
NEXT_PUBLIC_BACKEND_API_URL="https://app.joinlita.com" \
LITA_CPU_LIMIT=90 LITA_LOAD_REQUESTS=40 bash scripts/lita_smoke_test.sh

NEXT_PUBLIC_BACKEND_API_URL="https://app.joinlita.com" \
BACKEND_API_URL="https://app.joinlita.com" \
bash scripts/prelaunch_frontend_smoke_test.sh

Operational inputs used by the smoke checks

Variable Purpose
NEXT_PUBLIC_BACKEND_API_URL Points the frontend smoke run at the backend URL under test.
BACKEND_API_URL Backend URL used by the frontend smoke script.
LITA_CPU_LIMIT CPU threshold used by the load guard.
LITA_LOAD_REQUESTS Number of requests used in the short load burst.

Pre-launch sequence

sequenceDiagram
    participant Op as Operator
    participant WF as GitHub Actions workflow
    participant BE as FastAPI backend
    participant FE as Next.js frontend

    Op->>WF: Run DWILAR Pre Launch Stability
    WF->>BE: Execute lita_smoke_test.sh
    WF->>FE: Execute prelaunch_frontend_smoke_test.sh
    WF->>WF: Run short load burst and CPU guard
    WF-->>Op: Pass or fail result

Observability and Runtime Telemetry

The frontend exposes a small but explicit observability bootstrap through frontend/instrumentation.ts. That file selects a service name, registers OpenTelemetry, and loads Sentry configuration based on the runtime environment.

Telemetry bootstrap

frontend/instrumentation.ts

Element Type Purpose
serviceName constant Uses OTEL_SERVICE_NAME or falls back to lita-frontend.
register exported function Registers telemetry and loads runtime-specific Sentry config.

Public methods

Method Description
register Registers OpenTelemetry and conditionally loads Sentry configuration for the current runtime.

Runtime behavior

Condition Behavior
process.env.NEXT_RUNTIME === "nodejs" Imports ./sentry.server.config.
process.env.NEXT_RUNTIME === "edge" Imports ./sentry.edge.config.
Any runtime Calls registerOTel({ serviceName }).

Observability flow

sequenceDiagram
    participant Next as Next.js runtime
    participant Inst as instrumentation ts
    participant OTel as OpenTelemetry
    participant Sentry as Sentry config

    Next->>Inst: register
    Inst->>OTel: registerOTel serviceName
    alt nodejs runtime
        Inst->>Sentry: import sentry server config
    else edge runtime
        Inst->>Sentry: import sentry edge config
    end

Operational logging surface

The deployment guide uses two log inspection paths after rollout:

gcloud compute ssh instance-20250206-052936
cd MVP_fastapi
docker-compose -f platform/deployment/prod/docker-compose.yaml logs backend
docker-compose -f platform/deployment/prod/docker-compose.yml logs frontend

These commands are the documented path for checking container logs after deployment and for verifying that the backend and frontend started cleanly on the production host.

Deployment Process

The documented release process is intentionally linear: run tests, publish images, update the database schema when required, and roll the stack on the Compute Engine host. The schema step is part of the deployment guide because new API endpoints may depend on fresh tables before they can serve traffic.

Release phases

Phase Outcome
Testing Backend unit tests must pass before image publishing.
Publishing Backend and frontend images are pushed to Artifact Registry.
Schema update Long-lived databases are updated when the release introduces new tables.
Rollout The production host fetches the new images and restarts the stack.
Verification Operators confirm the application responds and logs are clean.

Operational verification checklist

  • Confirm the app is reachable at https://app.joinlita.com.
  • Confirm client-side routes continue to work after deployment.
  • Confirm /api/... requests are routed to the backend through the production edge path.
  • Inspect GCP health checks and container logs if the rollout behaves unexpectedly.

Error Handling

The documented workflow fails fast at each stage instead of attempting a partial release. Test failures stop publishing, registry authentication failures prevent image pushes, and SSH rollout failures stop the remote script before the new stack can be considered live.

Failure point Result
pytest tests/ -v fails Images are not published.
Artifact Registry auth fails Build artifacts cannot be pushed or pulled.
appleboy/ssh-action fails The Compute Engine rollout does not start.
docker-compose pull fails The host cannot fetch the new release images.
docker-compose up -d --build fails The production stack does not restart successfully.

Dependencies

The deployment stack depends on these concrete tools and services from the repository documentation:

  • GitHub Actions
  • docker/build-push-action@v4
  • appleboy/[email protected]
  • Google Cloud Artifact Registry
  • Google Compute Engine
  • gcloud auth configure-docker
  • docker-compose
  • pytest
  • Node.js 20 on the frontend image
  • OpenTelemetry and Sentry for frontend observability

Testing Considerations

The deployment and operations workflow is validated through the following concrete checks:

  • Backend unit tests: pytest tests/ -v
  • Scheduled pre-launch smoke workflow: DWILAR | Pre-Launch Stability
  • Backend smoke script: scripts/lita_smoke_test.sh
  • Frontend smoke script: scripts/prelaunch_frontend_smoke_test.sh
  • Production rollout verification via docker-compose logs
  • Runtime telemetry initialization via frontend/instrumentation.ts

Key Classes Reference

Class Responsibility
documents/deployment.md Canonical deployment, publishing, rollout, and verification workflow.
frontend/Dockerfile Production multi-stage frontend image build.
frontend/Dockerfile.local Local hot-reload frontend container.
frontend/instrumentation.ts Frontend telemetry bootstrap for OpenTelemetry and Sentry.
frontend/.env.local.example Frontend telemetry and runtime configuration surface.