Deployment, Operations, and Observability - Pre-launch Stability Checks, CPU and Retry Guards, and Canary Rollback Tooling

Overview

This release-control area gates launch on preflight stability checks and protects live rollouts with CPU and retry safeguards. The tooling is centered on .github/workflows/prelaunch.yml, which coordinates smoke tests and guard checks before a release can proceed.

The same operational bundle also supports canary deployment monitoring. scripts/check_service_cpu_anomalies.py, scripts/monitor_cpu_and_retries.sh, and scripts/canary_deployment.sh use scripts/service_cpu_baselines.json to compare observed service behavior against expected CPU levels and to trigger rollback when elevated load or retry behavior persists.

Stability and Rollback Architecture

flowchart TB
    subgraph PrelaunchGate [Prelaunch Gate]
        PrelaunchWorkflow[prelaunch workflow]
        BackendSmoke[lita_smoke_test.sh]
        FrontendSmoke[prelaunch_frontend_smoke_test.sh]
        CpuRetryTest[test_cpu_and_retry_prevention.py]
    end

    subgraph ObservabilityGuardrails [CPU and Retry Guardrails]
        CpuAnomalyCheck[check_service_cpu_anomalies.py]
        CpuMonitor[monitor_cpu_and_retries.sh]
        CpuBaselines[service_cpu_baselines.json]
    end

    subgraph CanaryControl [Canary Control]
        CanaryDeploy[canary_deployment.sh]
    end

    PrelaunchWorkflow --> BackendSmoke
    PrelaunchWorkflow --> FrontendSmoke
    PrelaunchWorkflow --> CpuRetryTest

    BackendSmoke --> PrelaunchWorkflow
    FrontendSmoke --> PrelaunchWorkflow
    CpuRetryTest --> PrelaunchWorkflow

    CpuBaselines --> CpuAnomalyCheck
    CpuBaselines --> CpuMonitor

    CanaryDeploy --> CpuMonitor
    CpuMonitor --> CpuAnomalyCheck
    CpuAnomalyCheck --> CanaryDeploy
    CanaryDeploy --> PrelaunchWorkflow

Prelaunch Workflow

.github/workflows/prelaunch.yml

File path: .github/workflows/prelaunch.yml

This workflow is the release gate for pre-launch stability checks. It orchestrates the smoke-test sequence, then runs CPU and retry prevention checks before release proceeds.

Attribute Detail
Runtime GitHub Actions
Role Prelaunch orchestration
Primary inputs scripts/lita_smoke_test.sh, scripts/prelaunch_frontend_smoke_test.sh, scripts/test_cpu_and_retry_prevention.py
Primary outputs Pass or fail outcome for the release gate
Failure behavior Stops the release path when a step exits non-zero

The workflow is designed to catch regressions before they reach a canary or production rollout. Its checks focus on two release risks: broken frontend-backend reachability and resource/retry regressions that can destabilize the services during launch.

Smoke Test Sequencing

Backend Smoke Test

File path: scripts/lita_smoke_test.sh

This script performs the backend side of the prelaunch smoke sequence. It exists to validate that the core service path responds before the release gate advances to frontend reachability checks and CPU/retry validation.

Attribute Detail
Runtime Shell
Role Backend smoke validation
Consumed by .github/workflows/prelaunch.yml
Guard focus Backend reachability and basic service health

Frontend Smoke Test

File path: scripts/prelaunch_frontend_smoke_test.sh

This script validates the frontend-to-backend release path. It is the smoke-test step that verifies the frontend can reach the backend through the deployment configuration used for launch.

Attribute Detail
Runtime Shell
Role Frontend smoke validation
Consumed by .github/workflows/prelaunch.yml
Guard focus Frontend-backend reachability

Smoke-Test Flow

sequenceDiagram
    participant GA as GitHub Actions
    participant WF as prelaunch workflow
    participant BS as lita smoke test shell
    participant FS as frontend smoke test shell
    participant PR as backend and frontend services
    participant CP as CPU and retry test

    GA->>WF: start prelaunch gate
    WF->>BS: run backend smoke test
    BS->>PR: validate backend reachability
    PR-->>BS: service response
    BS-->>WF: pass or fail

    WF->>FS: run frontend smoke test
    FS->>PR: validate frontend to backend reachability
    PR-->>FS: service response
    FS-->>WF: pass or fail

    WF->>CP: run CPU and retry prevention check
    CP-->>WF: pass or fail
    WF-->>GA: release gate result

The smoke sequence is intentionally staged: backend reachability is checked before frontend connectivity, and the workflow only advances to CPU and retry validation after the service path proves reachable. A failure at any step stops the gate.

CPU and Retry Guards

scripts/test_cpu_and_retry_prevention.py

File path: scripts/test_cpu_and_retry_prevention.py

This Python test validates two release-risk conditions: service CPU behavior versus baselines and retry-loop prevention. It is the prelaunch check that protects the release from shipping a regression that causes repeated request churn or excessive CPU use.

Attribute Detail
Runtime Python
Role CPU and retry regression test
Consumed by .github/workflows/prelaunch.yml
Guard focus CPU anomaly detection and retry-loop prevention
Baseline source scripts/service_cpu_baselines.json

scripts/check_service_cpu_anomalies.py

File path: scripts/check_service_cpu_anomalies.py

This script compares observed service CPU behavior against the configured baselines. It acts as the anomaly detector used by the operational guardrail path and supports both release checks and live monitoring.

Attribute Detail
Runtime Python
Role CPU anomaly checker
Consumed by scripts/monitor_cpu_and_retries.sh, release validation flows
Guard focus CPU deviation from service baseline

scripts/service_cpu_baselines.json

File path: scripts/service_cpu_baselines.json

This JSON asset stores the per-service CPU baselines used by the anomaly and monitoring scripts. It provides the expected comparison values for services under release or canary observation.

Attribute Detail
Format JSON
Role Baseline store
Consumed by scripts/check_service_cpu_anomalies.py, scripts/monitor_cpu_and_retries.sh
Content Per-service CPU baseline values

CPU and Retry Guard Flow

sequenceDiagram
    participant WF as prelaunch workflow
    participant PT as test_cpu_and_retry_prevention py
    participant CP as check_service_cpu_anomalies py
    participant BL as service_cpu_baselines json
    participant SV as services

    WF->>PT: run CPU and retry prevention test
    PT->>BL: load baseline values
    PT->>SV: inspect service CPU and retry behavior
    SV-->>PT: observed metrics
    PT->>CP: compare against anomaly thresholds
    CP->>BL: read baseline rules
    CP-->>PT: anomaly result
    PT-->>WF: pass or fail

These checks are meant to catch regressions before release by rejecting CPU growth patterns that diverge from the stored baselines and by rejecting retry behavior that can amplify load.

Canary Monitoring and Rollback

scripts/monitor_cpu_and_retries.sh

File path: scripts/monitor_cpu_and_retries.sh

This shell monitor watches live service CPU and retry activity during canary execution. It uses sustained-threshold logic rather than a single sample decision, so transient spikes do not have to be treated the same as repeated overload.

Attribute Detail
Runtime Shell
Role Live monitoring loop
Consumed by Canary deployment tooling
Guard focus Sustained CPU elevation and retry growth
Baseline source scripts/service_cpu_baselines.json

scripts/canary_deployment.sh

File path: scripts/canary_deployment.sh

This is the canary rollout controller. It performs the rollout step, watches the guard signals, and triggers rollback when the sustained-threshold logic indicates that the canary is unhealthy.

Attribute Detail
Runtime Shell
Role Canary deployment and rollback control
Consumed by Release and operations workflows
Guard focus Rollout promotion versus rollback
Inputs Live monitoring results from scripts/monitor_cpu_and_retries.sh

Sustained-Threshold Rollback Flow

sequenceDiagram
    participant CD as canary deployment shell
    participant MN as monitor cpu and retries shell
    participant CH as check service cpu anomalies py
    participant BL as service cpu baselines json
    participant SV as canary services

    CD->>MN: start monitoring window
    MN->>BL: load baseline values
    MN->>SV: read CPU and retry signals
    SV-->>MN: live observations
    MN->>CH: evaluate anomalies
    CH->>BL: compare against thresholds
    CH-->>MN: anomaly status
    MN-->>CD: sustained threshold result

    alt threshold sustained
        CD->>CD: trigger rollback
    else within baseline
        CD->>CD: continue canary progression
    end

The rollback logic is meant to catch regressions during release, not just before release. A canary can begin cleanly and still be rolled back if the same CPU or retry issue persists long enough to cross the sustained threshold.

Operational State Control

Release and Canary States

State Description
PrelaunchPending The workflow has started but no guard result has been accepted yet
SmokeTesting Backend and frontend reachability checks are running
CpuRetryValidation CPU and retry prevention logic is being evaluated
CanaryDeploying The canary rollout is active
Monitoring Live CPU and retry observation is in progress
Stable The canary remains within threshold and can continue
RollbackTriggered Sustained thresholds were exceeded and rollback was initiated
Failed A guard step returned a non-zero result or detected a violation
stateDiagram-v2
    [*] --> PrelaunchPending
    PrelaunchPending --> SmokeTesting
    SmokeTesting --> CpuRetryValidation
    CpuRetryValidation --> CanaryDeploying
    CanaryDeploying --> Monitoring
    Monitoring --> Stable
    Monitoring --> RollbackTriggered
    SmokeTesting --> Failed
    CpuRetryValidation --> Failed
    CanaryDeploying --> Failed
    RollbackTriggered --> [*]
    Stable --> [*]
    Failed --> [*]

Error Handling

The operational scripts use exit status as the primary failure signal. A failed smoke test, a CPU anomaly, or a retry-loop prevention failure stops the gate and prevents promotion to the next stage. During canary execution, sustained threshold violations switch the control path from rollout progression to rollback.

Integration Points

  • GitHub Actions release gating via .github/workflows/prelaunch.yml
  • Backend service reachability checks via scripts/lita_smoke_test.sh
  • Frontend-to-backend validation via scripts/prelaunch_frontend_smoke_test.sh
  • CPU baseline comparison via scripts/check_service_cpu_anomalies.py
  • Retry-loop prevention via scripts/test_cpu_and_retry_prevention.py
  • Live canary observation via scripts/monitor_cpu_and_retries.sh
  • Canary promotion and rollback control via scripts/canary_deployment.sh
  • Baseline thresholds stored in scripts/service_cpu_baselines.json

Testing Considerations

  • Backend smoke passes but frontend smoke fails: the release gate should fail before deployment proceeds.
  • CPU stays below the stored baseline while retries remain bounded: the prelaunch guard should pass.
  • CPU exceeds baseline only briefly: monitoring logic should rely on sustained-threshold behavior rather than a single sample.
  • CPU or retry growth persists across the observation window: canary rollback should be triggered.
  • Retry-loop prevention detects repeated request churn: the release gate should stop before rollout promotion.

Dependencies

  • GitHub Actions for workflow orchestration
  • Bash for smoke, monitoring, and canary scripts
  • Python for anomaly detection and retry prevention logic
  • scripts/service_cpu_baselines.json as the threshold store

Key Classes Reference

Class Responsibility
.github/workflows/prelaunch.yml Prelaunch release gate and step orchestration
scripts/lita_smoke_test.sh Backend smoke validation
scripts/prelaunch_frontend_smoke_test.sh Frontend-backend reachability validation
scripts/test_cpu_and_retry_prevention.py CPU and retry regression check
scripts/check_service_cpu_anomalies.py CPU anomaly evaluation against baselines
scripts/monitor_cpu_and_retries.sh Live monitoring with sustained-threshold tracking
scripts/canary_deployment.sh Canary deployment control and rollback trigger
scripts/service_cpu_baselines.json Per-service CPU baseline definitions