Deployment, Operations, and Observability - Pre-launch Stability Checks, CPU and Retry Guards, and Canary Rollback Tooling
Overview
This release-control area gates launch on preflight stability checks and protects live rollouts with CPU and retry safeguards. The tooling is centered on .github/workflows/prelaunch.yml, which coordinates smoke tests and guard checks before a release can proceed.
The same operational bundle also supports canary deployment monitoring. scripts/check_service_cpu_anomalies.py, scripts/monitor_cpu_and_retries.sh, and scripts/canary_deployment.sh use scripts/service_cpu_baselines.json to compare observed service behavior against expected CPU levels and to trigger rollback when elevated load or retry behavior persists.
Stability and Rollback Architecture
flowchart TB
subgraph PrelaunchGate [Prelaunch Gate]
PrelaunchWorkflow[prelaunch workflow]
BackendSmoke[lita_smoke_test.sh]
FrontendSmoke[prelaunch_frontend_smoke_test.sh]
CpuRetryTest[test_cpu_and_retry_prevention.py]
end
subgraph ObservabilityGuardrails [CPU and Retry Guardrails]
CpuAnomalyCheck[check_service_cpu_anomalies.py]
CpuMonitor[monitor_cpu_and_retries.sh]
CpuBaselines[service_cpu_baselines.json]
end
subgraph CanaryControl [Canary Control]
CanaryDeploy[canary_deployment.sh]
end
PrelaunchWorkflow --> BackendSmoke
PrelaunchWorkflow --> FrontendSmoke
PrelaunchWorkflow --> CpuRetryTest
BackendSmoke --> PrelaunchWorkflow
FrontendSmoke --> PrelaunchWorkflow
CpuRetryTest --> PrelaunchWorkflow
CpuBaselines --> CpuAnomalyCheck
CpuBaselines --> CpuMonitor
CanaryDeploy --> CpuMonitor
CpuMonitor --> CpuAnomalyCheck
CpuAnomalyCheck --> CanaryDeploy
CanaryDeploy --> PrelaunchWorkflow
Prelaunch Workflow
.github/workflows/prelaunch.yml
File path: .github/workflows/prelaunch.yml
This workflow is the release gate for pre-launch stability checks. It orchestrates the smoke-test sequence, then runs CPU and retry prevention checks before release proceeds.
| Attribute | Detail |
|---|---|
| Runtime | GitHub Actions |
| Role | Prelaunch orchestration |
| Primary inputs | scripts/lita_smoke_test.sh, scripts/prelaunch_frontend_smoke_test.sh, scripts/test_cpu_and_retry_prevention.py |
| Primary outputs | Pass or fail outcome for the release gate |
| Failure behavior | Stops the release path when a step exits non-zero |
The workflow is designed to catch regressions before they reach a canary or production rollout. Its checks focus on two release risks: broken frontend-backend reachability and resource/retry regressions that can destabilize the services during launch.
Smoke Test Sequencing
Backend Smoke Test
File path: scripts/lita_smoke_test.sh
This script performs the backend side of the prelaunch smoke sequence. It exists to validate that the core service path responds before the release gate advances to frontend reachability checks and CPU/retry validation.
| Attribute | Detail |
|---|---|
| Runtime | Shell |
| Role | Backend smoke validation |
| Consumed by | .github/workflows/prelaunch.yml |
| Guard focus | Backend reachability and basic service health |
Frontend Smoke Test
File path: scripts/prelaunch_frontend_smoke_test.sh
This script validates the frontend-to-backend release path. It is the smoke-test step that verifies the frontend can reach the backend through the deployment configuration used for launch.
| Attribute | Detail |
|---|---|
| Runtime | Shell |
| Role | Frontend smoke validation |
| Consumed by | .github/workflows/prelaunch.yml |
| Guard focus | Frontend-backend reachability |
Smoke-Test Flow
sequenceDiagram
participant GA as GitHub Actions
participant WF as prelaunch workflow
participant BS as lita smoke test shell
participant FS as frontend smoke test shell
participant PR as backend and frontend services
participant CP as CPU and retry test
GA->>WF: start prelaunch gate
WF->>BS: run backend smoke test
BS->>PR: validate backend reachability
PR-->>BS: service response
BS-->>WF: pass or fail
WF->>FS: run frontend smoke test
FS->>PR: validate frontend to backend reachability
PR-->>FS: service response
FS-->>WF: pass or fail
WF->>CP: run CPU and retry prevention check
CP-->>WF: pass or fail
WF-->>GA: release gate result
The smoke sequence is intentionally staged: backend reachability is checked before frontend connectivity, and the workflow only advances to CPU and retry validation after the service path proves reachable. A failure at any step stops the gate.
CPU and Retry Guards
scripts/test_cpu_and_retry_prevention.py
File path: scripts/test_cpu_and_retry_prevention.py
This Python test validates two release-risk conditions: service CPU behavior versus baselines and retry-loop prevention. It is the prelaunch check that protects the release from shipping a regression that causes repeated request churn or excessive CPU use.
| Attribute | Detail |
|---|---|
| Runtime | Python |
| Role | CPU and retry regression test |
| Consumed by | .github/workflows/prelaunch.yml |
| Guard focus | CPU anomaly detection and retry-loop prevention |
| Baseline source | scripts/service_cpu_baselines.json |
scripts/check_service_cpu_anomalies.py
File path: scripts/check_service_cpu_anomalies.py
This script compares observed service CPU behavior against the configured baselines. It acts as the anomaly detector used by the operational guardrail path and supports both release checks and live monitoring.
| Attribute | Detail |
|---|---|
| Runtime | Python |
| Role | CPU anomaly checker |
| Consumed by | scripts/monitor_cpu_and_retries.sh, release validation flows |
| Guard focus | CPU deviation from service baseline |
scripts/service_cpu_baselines.json
File path: scripts/service_cpu_baselines.json
This JSON asset stores the per-service CPU baselines used by the anomaly and monitoring scripts. It provides the expected comparison values for services under release or canary observation.
| Attribute | Detail |
|---|---|
| Format | JSON |
| Role | Baseline store |
| Consumed by | scripts/check_service_cpu_anomalies.py, scripts/monitor_cpu_and_retries.sh |
| Content | Per-service CPU baseline values |
CPU and Retry Guard Flow
sequenceDiagram
participant WF as prelaunch workflow
participant PT as test_cpu_and_retry_prevention py
participant CP as check_service_cpu_anomalies py
participant BL as service_cpu_baselines json
participant SV as services
WF->>PT: run CPU and retry prevention test
PT->>BL: load baseline values
PT->>SV: inspect service CPU and retry behavior
SV-->>PT: observed metrics
PT->>CP: compare against anomaly thresholds
CP->>BL: read baseline rules
CP-->>PT: anomaly result
PT-->>WF: pass or fail
These checks are meant to catch regressions before release by rejecting CPU growth patterns that diverge from the stored baselines and by rejecting retry behavior that can amplify load.
Canary Monitoring and Rollback
scripts/monitor_cpu_and_retries.sh
File path: scripts/monitor_cpu_and_retries.sh
This shell monitor watches live service CPU and retry activity during canary execution. It uses sustained-threshold logic rather than a single sample decision, so transient spikes do not have to be treated the same as repeated overload.
| Attribute | Detail |
|---|---|
| Runtime | Shell |
| Role | Live monitoring loop |
| Consumed by | Canary deployment tooling |
| Guard focus | Sustained CPU elevation and retry growth |
| Baseline source | scripts/service_cpu_baselines.json |
scripts/canary_deployment.sh
File path: scripts/canary_deployment.sh
This is the canary rollout controller. It performs the rollout step, watches the guard signals, and triggers rollback when the sustained-threshold logic indicates that the canary is unhealthy.
| Attribute | Detail |
|---|---|
| Runtime | Shell |
| Role | Canary deployment and rollback control |
| Consumed by | Release and operations workflows |
| Guard focus | Rollout promotion versus rollback |
| Inputs | Live monitoring results from scripts/monitor_cpu_and_retries.sh |
Sustained-Threshold Rollback Flow
sequenceDiagram
participant CD as canary deployment shell
participant MN as monitor cpu and retries shell
participant CH as check service cpu anomalies py
participant BL as service cpu baselines json
participant SV as canary services
CD->>MN: start monitoring window
MN->>BL: load baseline values
MN->>SV: read CPU and retry signals
SV-->>MN: live observations
MN->>CH: evaluate anomalies
CH->>BL: compare against thresholds
CH-->>MN: anomaly status
MN-->>CD: sustained threshold result
alt threshold sustained
CD->>CD: trigger rollback
else within baseline
CD->>CD: continue canary progression
end
The rollback logic is meant to catch regressions during release, not just before release. A canary can begin cleanly and still be rolled back if the same CPU or retry issue persists long enough to cross the sustained threshold.
Operational State Control
Release and Canary States
| State | Description |
|---|---|
| PrelaunchPending | The workflow has started but no guard result has been accepted yet |
| SmokeTesting | Backend and frontend reachability checks are running |
| CpuRetryValidation | CPU and retry prevention logic is being evaluated |
| CanaryDeploying | The canary rollout is active |
| Monitoring | Live CPU and retry observation is in progress |
| Stable | The canary remains within threshold and can continue |
| RollbackTriggered | Sustained thresholds were exceeded and rollback was initiated |
| Failed | A guard step returned a non-zero result or detected a violation |
stateDiagram-v2
[*] --> PrelaunchPending
PrelaunchPending --> SmokeTesting
SmokeTesting --> CpuRetryValidation
CpuRetryValidation --> CanaryDeploying
CanaryDeploying --> Monitoring
Monitoring --> Stable
Monitoring --> RollbackTriggered
SmokeTesting --> Failed
CpuRetryValidation --> Failed
CanaryDeploying --> Failed
RollbackTriggered --> [*]
Stable --> [*]
Failed --> [*]
Error Handling
The operational scripts use exit status as the primary failure signal. A failed smoke test, a CPU anomaly, or a retry-loop prevention failure stops the gate and prevents promotion to the next stage. During canary execution, sustained threshold violations switch the control path from rollout progression to rollback.
Integration Points
- GitHub Actions release gating via
.github/workflows/prelaunch.yml - Backend service reachability checks via
scripts/lita_smoke_test.sh - Frontend-to-backend validation via
scripts/prelaunch_frontend_smoke_test.sh - CPU baseline comparison via
scripts/check_service_cpu_anomalies.py - Retry-loop prevention via
scripts/test_cpu_and_retry_prevention.py - Live canary observation via
scripts/monitor_cpu_and_retries.sh - Canary promotion and rollback control via
scripts/canary_deployment.sh - Baseline thresholds stored in
scripts/service_cpu_baselines.json
Testing Considerations
- Backend smoke passes but frontend smoke fails: the release gate should fail before deployment proceeds.
- CPU stays below the stored baseline while retries remain bounded: the prelaunch guard should pass.
- CPU exceeds baseline only briefly: monitoring logic should rely on sustained-threshold behavior rather than a single sample.
- CPU or retry growth persists across the observation window: canary rollback should be triggered.
- Retry-loop prevention detects repeated request churn: the release gate should stop before rollout promotion.
Dependencies
- GitHub Actions for workflow orchestration
- Bash for smoke, monitoring, and canary scripts
- Python for anomaly detection and retry prevention logic
scripts/service_cpu_baselines.jsonas the threshold store
Key Classes Reference
| Class | Responsibility |
|---|---|
.github/workflows/prelaunch.yml |
Prelaunch release gate and step orchestration |
scripts/lita_smoke_test.sh |
Backend smoke validation |
scripts/prelaunch_frontend_smoke_test.sh |
Frontend-backend reachability validation |
scripts/test_cpu_and_retry_prevention.py |
CPU and retry regression check |
scripts/check_service_cpu_anomalies.py |
CPU anomaly evaluation against baselines |
scripts/monitor_cpu_and_retries.sh |
Live monitoring with sustained-threshold tracking |
scripts/canary_deployment.sh |
Canary deployment control and rollback trigger |
scripts/service_cpu_baselines.json |
Per-service CPU baseline definitions |