Deployment, Operations, and Observability
Overview
This section covers the release workflow for the MVP stack, from pre-launch validation to post-deploy verification and rollback-oriented checks. The documented process combines backend unit tests, local smoke scripts, and Docker Compose rollouts on a Compute Engine host, with Fastly and the Google Cloud HTTPS Load Balancer handling production traffic routing.
Operationally, the goal is to catch stability regressions before launch, validate that the frontend can still reach backend APIs after deployment, and give operators a repeatable way to inspect health and container logs on the host. The workflow also includes a CPU guard and short load burst during pre-launch validation so the release can fail fast on runaway behavior before it reaches production traffic.
Architecture Overview
flowchart TD
subgraph GitHubActions [GitHub Actions Workflow]
GA1[DWILAR Pre Launch Stability]
GA2[CPU guard]
GA3[Short load burst]
end
subgraph LocalValidation [Local Prelaunch Validation]
LS1[scripts lita_smoke_test.sh]
LS2[backend pytest]
LS3[uvicorn SQLite fallback]
FS1[scripts prelaunch_frontend_smoke_test.sh]
FS2[Next js app]
FS3[Frontend health route]
end
subgraph ProdHost [Compute Engine Host]
PH1[docker compose pull]
PH2[docker compose up]
PH3[backend container]
PH4[frontend container]
PH5[container logs]
end
GA1 --> LS1
GA1 --> GA2
GA1 --> GA3
LS1 --> LS2
LS2 --> LS3
LS3 --> FS1
FS1 --> FS2
FS2 --> FS3
GA1 --> PH1
PH1 --> PH2
PH2 --> PH3
PH2 --> PH4
PH3 --> PH5
PH4 --> PH5
Pre-launch Stability Workflow
The documented pre-launch workflow is the DWILAR | Pre-Launch Stability GitHub Actions job. It runs the backend smoke path first, then exercises the frontend startup and health check path. The workflow is designed to run against a local FastAPI instance with the SQLite fallback, which keeps the validation loop fast and isolated from production databases.
The pre-launch job also includes a short load burst and CPU guard. The documented local inputs are LITA_CPU_LIMIT=90 and LITA_LOAD_REQUESTS=40, which let operators tune the stress check before release.
Run the workflow
- Open GitHub Actions → DWILAR | Pre-Launch Stability.
- Click Run workflow on the target branch.
- Optionally set the
backend_urlinput to the environment you want to validate.
Local validation commands
Run the same smoke steps locally when you need a quick pre-flight check from the repo root:
python -m pip install -r backend/requirements.txt
NEXT_PUBLIC_BACKEND_API_URL="https://app.joinlita.com" \
LITA_CPU_LIMIT=90 LITA_LOAD_REQUESTS=40 bash scripts/lita_smoke_test.sh
NEXT_PUBLIC_BACKEND_API_URL="https://app.joinlita.com" \
BACKEND_API_URL="https://app.joinlita.com" \
bash scripts/prelaunch_frontend_smoke_test.sh
Pre-launch sequence
sequenceDiagram
participant O as Operator
participant G as GitHub Actions
participant S as Smoke Script
participant P as Pytest
participant A as FastAPI App
participant F as Frontend Smoke Script
participant N as Next js App
participant B as Backend API
O->>G: Run DWILAR Pre Launch Stability
G->>S: Start lita_smoke_test.sh
S->>P: Run backend tests
P-->>S: Test results
S->>A: Start FastAPI with SQLite fallback
S->>A: Create consent and token
S->>A: Save and read snapshot
S->>A: Score twice for idempotency
A-->>S: Smoke responses
G->>F: Start prelaunch_frontend_smoke_test.sh
F->>N: Start Next js app
F->>B: Verify frontend health route can reach backend
B-->>F: Health response
F-->>G: Frontend smoke result
G-->>O: Pass or fail
Smoke Test Scripts
scripts/lita_smoke_test.sh
[!IMPORTANT] The backend smoke path is not just a startup check. It runs the targeted pytest module, starts FastAPI against the SQLite fallback, then exercises consent, token, snapshot, and scoring behavior before the frontend smoke path runs.
This script is the primary backend smoke harness referenced by the deployment documentation and Lita testing guide. It validates the backend test path first, then runs the application flow against a local FastAPI process using the SQLite fallback.
| Item | Details |
|---|---|
| Purpose | Validate backend readiness before launch |
| Test command | Targeted pytest module for the Lita endpoints |
| Runtime | Local FastAPI instance with SQLite fallback |
| Smoke actions | Create consent, issue token, save snapshot, read snapshot, score twice |
| Idempotency check | Repeats the scoring request with the same request_id |
| Dependency bootstrap | Automatically installs backend requirements unless disabled |
Key inputs and controls:
| Variable | Used For | Meaning |
|---|---|---|
LITA_BOOTSTRAP_DEPS |
Dependency bootstrap | Set to 0 to disable automatic pip install |
LITA_CPU_LIMIT |
Load guard | CPU ceiling used during the pre-launch load burst |
LITA_LOAD_REQUESTS |
Load guard | Number of requests used for the short stress run |
NEXT_PUBLIC_BACKEND_API_URL |
API target | Backend URL used by the smoke path |
The script is documented as a single-command verification path:
./scripts/lita_smoke_test.sh
scripts/prelaunch_frontend_smoke_test.sh
This script starts the frontend and verifies that the frontend health route can reach the backend. It is the frontend counterpart to the backend smoke harness and is explicitly referenced in the deployment notes.
| Item | Details |
|---|---|
| Purpose | Verify frontend startup and backend reachability |
| Runtime | Starts the Next.js app |
| Health check | Confirms the frontend health route can reach the backend |
| Inputs | Backend URL environment variables |
Key inputs:
| Variable | Used For | Meaning |
|---|---|---|
NEXT_PUBLIC_BACKEND_API_URL |
Frontend runtime | Backend URL exposed to the frontend |
BACKEND_API_URL |
Smoke script | Backend URL used by the verification script |
backend_url |
GitHub Actions workflow | Optional target environment for the pre-launch run |
Backend Validation with Pytest
The deployment guide explicitly calls for backend unit tests before deployment. The pre-launch smoke script also runs a targeted backend test module.
Documented commands
pytest tests/ -v
pytest backend/tests/interface/test_lita_endpoints.py -q
The first command is the deployment-level validation step. The second command is the targeted backend regression check used by the smoke workflow for consent, token, snapshot, and scoring coverage.
Validation scope
- Ensures the backend test suite passes before image rollout.
- Verifies the Lita API smoke flow against a local runtime.
- Confirms scoring idempotency by repeating the same
request_id.
Rollback-Oriented Checklist
The deployment instructions already define the canonical container restart sequence on the Compute Engine host. Use the same sequence as the base rollback path when a release needs to be backed out or rechecked.
Checklist
- SSH into the Compute Engine host:
gcloud compute ssh instance-20250206-052936
- Change into the deployment directory:
cd ~/MVP_fastapi/platform/deployment/prod
- Stop the current stack:
docker-compose down
- Re-authenticate Docker to Artifact Registry if needed:
gcloud auth configure-docker us-central1-docker.pkg.dev
- Pull the images for the checked-out revision:
docker-compose pull
- Bring the stack back up:
docker-compose up -d --build
- Check the frontend health route:
curl http://localhost:3000/api/ready
-
Confirm backend connectivity through the frontend proxy routes using a known application identifier from the deployed dataset.
-
Inspect logs if any route check fails.
Post-Deploy Verification and Observability
Post-deploy checks are a combination of browser access, API routing validation, load balancer health, and container log inspection.
What to verify
- The application is reachable at
https://app.joinlita.com. - Client-side routes such as
/loginand/registerstill render correctly. - Requests under
https://app.joinlita.com/api/...are routed correctly. - GCP Load Balancer health checks show healthy backend status.
- Backend and frontend container logs are clean enough to rule out startup or routing failures.
Container log inspection on the Compute Engine host
The deployment guide shows host-side log inspection with Docker Compose:
gcloud compute ssh instance-20250206-052936
cd MVP_fastapi
docker-compose -f platform/deployment/prod/docker-compose.yaml logs backend
docker-compose -f platform/deployment/prod/docker-compose.yml logs frontend
Observability checklist
[!NOTE] The documented log commands mix
docker-compose.yamlanddocker-compose.yml. The deploy step itself uses thedocker-compose.ymlpath, so copy the log command carefully and keep the same compose file name that is present on the host.
- Check Google Cloud Load Balancer health in the console.
- Inspect backend container logs for API or startup failures.
- Inspect frontend container logs for route or backend proxy failures.
- Re-run the frontend health route check after any container restart.
- If a verification fails, repeat the pull/up sequence after correcting the checked-out revision.
Operational API Checks
These routes are useful as post-deploy probes because they exercise the frontend-to-backend path and surface backend reachability problems.
Get Applicant Account Coverage
Get Applicant Account Coverage
{
"title": "Get Applicant Account Coverage",
"description": "Checks whether submitted connected accounts cover the accounts detected by Plaid and proxies the backend response through the frontend",
"method": "GET",
"baseUrl": "<FrontendBaseUrl>",
"endpoint": "/api/v1/applicants/{id}/account-coverage",
"headers": [],
"queryParams": [],
"pathParams": [
{
"key": "id",
"type": "string",
"required": true,
"description": "Applicant identifier"
}
],
"bodyType": "none",
"requestBody": "",
"formData": [],
"rawBody": "",
"responses": {
"200": {
"description": "Success",
"body": "{\n \"application_id\": 123,\n \"status\": \"incomplete\",\n \"reported_accounts\": 2,\n \"detected_accounts\": 3,\n \"missing_accounts\": 1,\n \"coverage_ratio\": 0.67,\n \"details\": [\n {\n \"connection_name\": \"Example Bank\",\n \"institution_name\": \"Example Bank\",\n \"reported_accounts\": 2,\n \"detected_accounts\": 3,\n \"missing_accounts\": 1,\n \"status\": \"incomplete\"\n }\n ],\n \"errors\": []\n}"
},
"400": {
"description": "Invalid applicant id",
"body": "{\n \"error\": \"Invalid applicant id.\"\n}"
},
"500": {
"description": "Backend API URL not configured",
"body": "{\n \"error\": \"Backend API URL is not configured. Set NEXT_PUBLIC_BACKEND_API_URL or BACKEND_API_URL.\"\n}"
},
"502": {
"description": "Backend unreachable",
"body": "{\n \"error\": \"Failed to reach backend API.\"\n}"
}
}
}
Get Applicant Transactions
Get Applicant Transactions
{
"title": "Get Applicant Transactions",
"description": "Returns transaction history for a selected applicant after combining backend, remote, and locally stored application data",
"method": "GET",
"baseUrl": "<FrontendBaseUrl>",
"endpoint": "/api/v1/applicants/{id}/transactions",
"headers": [],
"queryParams": [
{
"key": "start_date",
"type": "string",
"required": false,
"description": "Lower bound for the transaction date filter"
},
{
"key": "end_date",
"type": "string",
"required": false,
"description": "Upper bound for the transaction date filter"
}
],
"pathParams": [
{
"key": "id",
"type": "string",
"required": true,
"description": "Applicant identifier"
}
],
"bodyType": "none",
"requestBody": "",
"formData": [],
"rawBody": "",
"responses": {
"200": {
"description": "Success",
"body": "{\n \"applicant_id\": \"123\",\n \"start_date\": null,\n \"end_date\": null,\n \"transactions\": [\n {\n \"amount\": -42.5,\n \"date\": \"2024-05-01\",\n \"currency_code\": \"USD\",\n \"status\": \"posted\",\n \"account_name\": \"Checking\",\n \"connection_name\": \"Example Bank\",\n \"description\": \"Coffee shop\",\n \"categories\": [\n \"food_and_drink\",\n \"coffee_shop\"\n ],\n \"provider\": \"plaid\",\n \"metadata\": {\n \"amount\": -42.5,\n \"date\": \"2024-05-01\",\n \"currency_code\": \"USD\",\n \"status\": \"posted\",\n \"account_name\": \"Checking\",\n \"connection_name\": \"Example Bank\",\n \"description\": \"Coffee shop\",\n \"categories\": [\n \"food_and_drink\",\n \"coffee_shop\"\n ],\n \"provider\": \"plaid\"\n }\n }\n ]\n}"
},
"400": {
"description": "Invalid request",
"body": "{\n \"error\": \"Invalid applicant id.\"\n}"
},
"404": {
"description": "Application missing or no transaction history",
"body": "{\n \"error\": \"No transaction history is available for this applicant yet. Connect and sync bank accounts first.\"\n}"
},
"502": {
"description": "Backend unreachable",
"body": "{\n \"error\": \"Failed to reach backend API.\"\n}"
}
}
}
Dependencies
[!NOTE] The frontend proxy skips backend
404responses while trying base URL candidates. If every candidate returns404, the route can still end in a502style failure message instead of surfacing a plain not found response.
pytestfor backend validation.uvicornfor local FastAPI startup during smoke tests.- Docker and Docker Compose for the host rollout.
- Google Cloud CLI for Compute Engine SSH access and Artifact Registry auth.
- GitHub Actions for the pre-launch stability workflow.
- Google Cloud HTTPS Load Balancer health checks and console inspection.
- Fastly in front of the public application origin.
Testing Considerations
- Run the backend pytest path before any image rollout.
- Use the smoke scripts to validate both the backend flow and the frontend-to-backend path.
- Keep
LITA_CPU_LIMITandLITA_LOAD_REQUESTSaligned with the release environment when using the CPU guard. - Confirm the frontend health route passes before and after any compose restart.
- Inspect host-side logs any time a route check returns a
502or the frontend cannot reach backend data.
Key Classes Reference
| Class | Responsibility |
|---|---|
lita_smoke_test.sh |
Runs backend pytest, starts FastAPI on the SQLite fallback, and exercises consent, snapshot, and scoring smoke checks |
prelaunch_frontend_smoke_test.sh |
Starts the frontend and verifies the health route can reach the backend |
route.ts |
Proxies applicant account coverage checks through the frontend to the backend |
route.ts |
Combines backend and stored data to return applicant transaction history |
deployment.md |
Documents the release, post-deploy, and log inspection workflow |