Deployment, Operations, and Observability - ML Artifact Build, Vertex AI Deployment Verification, and Machine Retry Operations

Overview

This section covers the operator-facing flow for building the Vertex AI training image, pushing it to Artifact Registry, running the end-to-end pipeline, and verifying the deployed model before and after rollout. It is the fast reference for confirming that the training artifact, deployment target, and runtime prediction contract still line up after an image change or machine-type change.

The workflow is driven by deployment-level .env files and the helper scripts under scripts/vertex_ai/. The verification path is designed to work from the repo root through Docker Compose, while the retry policy inside run_vertex_pipeline.py automatically walks ordered machine-type lists when training or serving fails for memory-related reasons.

Architecture Overview

flowchart TB
    subgraph Config[Deployment Configuration]
        LocalEnv[platform deployment local env example]
        ProdEnv[platform deployment prod env example]
    end

    subgraph Auth[Registry Auth and Access]
        EnsureAuth[ensure_registry_and_auth sh]
        DockerAuth[gcloud docker configure]
        ArtifactRepo[Artifact Registry repo]
    end

    subgraph Build[Training Image Build]
        BuildPush[build_and_push_training_image sh]
        ComposeBuild[docker compose build]
        NormalizedUri[Normalized TRAIN_IMAGE_URI]
    end

    subgraph Orchestration[Vertex Orchestration]
        RunCompose[run_pipeline_via_compose sh]
        FastCheck[fast_credit_pipeline_check sh]
        VerifyRuntime[verify_runtime_credit_scoring sh]
        PipelineModule[run_vertex_pipeline module]
    end

    subgraph VertexAI[Vertex AI]
        CustomJob[Custom Job]
        ModelUpload[Model upload]
        EndpointDeploy[Endpoint deploy]
        PredictCall[Endpoint predict]
    end

    subgraph Retry[Retry Policy]
        TrainMachines[TRAIN_MACHINE_TYPES]
        DeployMachines[DEPLOY_MACHINE_TYPES]
        MemoryAndQuota[Memory and quota errors]
    end

    LocalEnv --> EnsureAuth
    ProdEnv --> EnsureAuth
    EnsureAuth --> DockerAuth
    EnsureAuth --> ArtifactRepo
    BuildPush --> EnsureAuth
    BuildPush --> ComposeBuild
    ComposeBuild --> NormalizedUri
    RunCompose --> PipelineModule
    RunCompose --> CustomJob
    RunCompose --> ModelUpload
    RunCompose --> EndpointDeploy
    RunCompose --> PredictCall
    FastCheck --> VerifyRuntime
    VerifyRuntime --> PredictCall
    PipelineModule --> CustomJob
    PipelineModule --> ModelUpload
    PipelineModule --> EndpointDeploy
    PipelineModule --> PredictCall
    TrainMachines --> CustomJob
    DeployMachines --> EndpointDeploy
    MemoryAndQuota --> TrainMachines
    MemoryAndQuota --> DeployMachines

Deployment Configuration

Deployment level env examples

platform/deployment/local/.env.example

platform/deployment/prod/.env.example

These starter files are referenced by the verification guide and are the primary place operators are expected to set Vertex AI training and deployment values.

Variable Type Purpose
PROJECT_ID string GCP project that owns the Artifact Registry repo and Vertex AI resources
REGION string Vertex AI region, used to normalize Artifact Registry hostnames
TRAIN_IMAGE_URI string Training image tag used by Compose and Vertex AI Custom Jobs
DATA_URI string Training data source passed to the trainer
MODEL_OUTPUT_DIR string GCS path where model.joblib is written
PREDICTION_CONTAINER_IMAGE string Prebuilt Vertex prediction container used for model serving
GCP_CREDENTIALS_PATH string Path to the service account key used by the registry helper
SERVICE_ACCOUNT_EMAIL string Service account used for impersonation when a key is not supplied
DEPLOY_MACHINE_TYPE string Single-machine serving shape override
TRAIN_MACHINE_TYPE string Primary training machine type
TRAIN_MACHINE_TYPE_FALLBACK string Fallback training machine type
TRAIN_MACHINE_TYPES string list Ordered training retry list
DEPLOY_MACHINE_TYPES string list Ordered serving retry list
IMAGE_IMPORT_CHECK boolean Enables the preflight Docker import check
BUILD_BEFORE_RUN boolean Controls whether run_pipeline_via_compose.sh rebuilds first
PUSH_IMAGE_BEFORE_RUN boolean Controls whether the freshly built image is pushed before the pipeline run
RUN_TRAIN_STEP boolean Enables the training stage in run_vertex_pipeline.py
RUN_UPLOAD_STEP boolean Enables the model upload stage
RUN_DEPLOY_STEP boolean Enables endpoint deployment
RUN_PREDICT_STEP boolean Enables the prediction smoke test
FAIL_ON_PREDICT_ERROR boolean Converts predict smoke failures into hard failures when enabled
FAST_CHECK_REPORT_PATH string Output path for the machine-friendly fast-check report
VERIFY_STRICT boolean Makes runtime verification fail when traffic is not ready
WAIT_FOR_TRAFFIC_SPLIT_SECONDS integer Wait window for endpoint traffic readiness
VERIFY_ALLOW_SHAPE_MISMATCH boolean Allows known serving-contract mismatch warnings to pass
AUTO_SELECT_DEPLOYED_ENDPOINT boolean Auto-selects a recent endpoint with active traffic
EXISTING_ENDPOINT_NAME string Used when running predict-only mode against an already deployed endpoint
CREDIT_SCORE_PROVIDER string Runtime provider selector for deployed Vertex scoring
VERTEX_CREDIT_ENDPOINT_NAME string Runtime endpoint name for Vertex credit scoring
VERTEX_CREDIT_ENDPOINT_ID string Runtime endpoint id for Vertex credit scoring
CREDIT_SCORE_ALLOW_HEURISTIC_FALLBACK boolean Controls fail-closed behavior in runtime scoring

Command Entry Points

scripts/vertex_ai/ensure_registry_and_auth.sh

scripts/vertex_ai/build_and_push_training_image.sh

scripts/vertex_ai/run_pipeline_via_compose.sh

scripts/vertex_ai/fast_credit_pipeline_check.sh

scripts/vertex_ai/verify_runtime_credit_scoring.sh

Script Responsibility
ensure_registry_and_auth.sh Activates registry credentials, configures Docker auth for Artifact Registry, creates the repo if missing, and grants writer access to the service account
build_and_push_training_image.sh Wraps registry setup, normalizes the training image host to the active region, builds the training image, and pushes it
run_pipeline_via_compose.sh Runs the training, upload, deploy, and smoke-predict pipeline through Compose
fast_credit_pipeline_check.sh Performs a quick training-image import check and then runs runtime verification
verify_runtime_credit_scoring.sh Verifies the live endpoint by running a prediction through the deployed credit-scoring path

Training image push prerequisites

Before pushing the training image, the guide requires:

  1. PROJECT_ID set in the deployment env file.
  2. gcloud auth login
  3. gcloud auth application-default login
  4. gcloud config set project "$PROJECT_ID"
  5. gcloud auth configure-docker us-central1-docker.pkg.dev
  6. Either:
    • a service account key at ./service-key.json, or
    • impersonation of SERVICE_ACCOUNT_EMAIL

The registry helper also creates the Artifact Registry repository if it does not exist and applies the writer grant needed for pushes to succeed.

Build platform requirement

[!IMPORTANT] The helpers normalize TRAIN_IMAGE_URI so the hostname matches REGION. This avoids pull failures when the env file still points at an older region host.

The training image build is forced to linux/amd64 during Compose build so the pushed image matches the platform expected by Vertex AI workers. The verification guide explicitly calls out this requirement to prevent manifest platform mismatches on Apple Silicon and other non-amd64 hosts.

Vertex Pipeline Orchestration Module

backend/src/credit-score-pipeline/run_vertex_pipeline.py

This module is the execution core behind the Vertex AI flow. The tests in backend/tests/usecase/test_vertex_pipeline_machine_retry.py show that it supports stage flags, machine-type retries, existing-endpoint predict-only runs, and shape-aware prediction retries.

Module properties

Property Type Description
TRAIN_MACHINE_TYPES list[str] Ordered machine-type candidates for training retries
DEPLOY_MACHINE_TYPES list[str] Ordered machine-type candidates for deployment retries
RUN_TRAIN_STEP bool Enables or disables the training stage
RUN_UPLOAD_STEP bool Enables or disables the model upload stage
RUN_DEPLOY_STEP bool Enables or disables the deployment stage
RUN_PREDICT_STEP bool Enables or disables the prediction smoke test
FAIL_ON_PREDICT_ERROR bool Controls whether predict smoke failures abort the run
EXISTING_ENDPOINT_NAME str or None Required for predict-only verification against an existing endpoint

Public methods

Method Description
run_training_job Creates and runs the Vertex AI Custom Job, retrying across training machine types when memory-related errors occur
upload_model Uploads the trained artifact for deployment
deploy_model Creates an endpoint and deploys the model, retrying across serving machine types when memory or quota errors occur
predict_example Runs the smoke prediction against the deployed endpoint and retries shape-sensitive payloads when needed
resolve_existing_model Resolves the model for predict-only operation
resolve_existing_endpoint Resolves the endpoint for predict-only operation
main Executes the configured pipeline stages in order
_augment_training_error Rewrites low-level trainer failures into operator-readable errors
_is_memory_error Detects memory-constrained training and serving failures
_is_instance_type_error Detects dict payload rejection during prediction
_is_feature_count_mismatch_error Detects feature-shape mismatch failures
_expected_feature_count Extracts the expected feature count from a mismatch error
_resize_numeric_instance Pads or truncates numeric vectors to a target length
_to_numeric_instance Converts mixed prediction inputs to a numeric list

Stage initialization flow

  1. The module reads the stage flags and machine-type lists from env-backed constants.
  2. If RUN_TRAIN_STEP is enabled, it calls run_training_job.
  3. If RUN_UPLOAD_STEP is enabled, it uploads the resulting model.
  4. If RUN_DEPLOY_STEP is enabled, it deploys the model to Vertex AI.
  5. If RUN_PREDICT_STEP is enabled, it runs a smoke prediction.
  6. If the predict step fails and FAIL_ON_PREDICT_ERROR is false, the pipeline logs the failure and continues.

Data fetching and runtime verification flow

The predict path is intentionally contract-aware:

  • it starts with a dictionary-style instance,
  • retries with a numeric vector when the endpoint rejects the dict payload,
  • retries again with a resized numeric vector when the endpoint reports a feature-count mismatch.

That behavior is verified by the unit tests that assert a three-step progression from dict payload to numeric list to resized numeric list.

Training Image Build and Registry Auth

build_and_push_training_image.sh

This helper is the entry point for image publishing. It wraps registry setup, normalizes the image URI host to the active region, and ensures the image that Vertex pulls is the same image Compose just built.

ensure_registry_and_auth.sh

This helper handles the registry access prerequisites for pushes:

  • uses the service key at GCP_CREDENTIALS_PATH when present,
  • otherwise impersonates SERVICE_ACCOUNT_EMAIL,
  • configures Docker for the Artifact Registry host derived from TRAIN_IMAGE_URI,
  • creates the Artifact Registry repo if it is missing,
  • grants writer access to the service account so pushes succeed.

Build and push flow

sequenceDiagram
    participant Ops as Operator
    participant Build as build_and_push_training_image sh
    participant Auth as ensure_registry_and_auth sh
    participant Gcloud as gcloud
    participant Docker as Docker
    participant Repo as Artifact Registry

    Ops->>Build: run with deployment env file
    Build->>Auth: prepare registry access
    Auth->>Gcloud: activate key or impersonate service account
    Auth->>Gcloud: configure docker auth
    Auth->>Repo: create repo if missing
    Auth->>Repo: grant writer access
    Build->>Docker: compose build
    Build->>Docker: compose push
    Build-->>Ops: normalized image pushed

Vertex AI Deployment Verification

The verification guide documents two related checks:

  1. a full orchestration run that trains, uploads, deploys, and runs a smoke prediction, and
  2. a fast repeatable smoke check that verifies the image import and endpoint runtime contract without retraining.

Verification semantics

Full pipeline run

run_pipeline_via_compose.sh is the operator path for the end-to-end job. It is documented to:

  • build the vertex-ai-pipeline service before running by default,
  • push the same image before the Custom Job starts,
  • submit the Custom Job,
  • upload the artifact,
  • deploy the endpoint,
  • run a test prediction,
  • log the CustomJob, model upload, endpoint, and prediction resource results.

Fast smoke check

fast_credit_pipeline_check.sh performs two checks:

  • a small sklearn import check using ExtraTreesClassifier to catch corrupted training images early,
  • runtime verification of the deployed endpoint.

The helper supports:

  • IMAGE_IMPORT_CHECK=0 to skip the image import check,
  • VERIFY_STRICT=1 to fail when no endpoint traffic is ready,
  • WAIT_FOR_TRAFFIC_SPLIT_SECONDS to tune how long it waits,
  • VERIFY_ALLOW_SHAPE_MISMATCH=0 to fail hard on serving-contract mismatches,
  • VERIFY_ALLOW_SHAPE_MISMATCH=1 to tolerate known legacy artifact issues with a warning.

When the warning path is used, the helper emits an explicit completed with WARNING summary and writes a final machine-readable status line.

Runtime endpoint verification

verify_runtime_credit_scoring.sh is the live endpoint verifier. The guide says it:

  • prefers the Python SDK google-cloud-aiplatform,
  • falls back to gcloud ai endpoints predict when the SDK is missing,
  • auto-selects a recently updated endpoint with active traffic when the configured endpoint has no traffic split yet,
  • waits briefly in non-strict mode,
  • waits longer and fails in strict mode when traffic never becomes ready.

Verification flow

sequenceDiagram
    participant Ops as Operator
    participant Check as fast_credit_pipeline_check sh
    participant Runtime as verify_runtime_credit_scoring sh
    participant SDK as Vertex SDK or gcloud
    participant Endpoint as Deployed Endpoint

    Ops->>Check: run fast smoke check
    Check->>Check: import training image
    Check->>Runtime: verify live endpoint
    Runtime->>SDK: predict request
    SDK->>Endpoint: endpoint predict
    Endpoint-->>SDK: predictions
    SDK-->>Runtime: result
    Runtime-->>Check: pass warn or fail
    Check-->>Ops: FAST_CHECK_RESULT

Success and failure signals

Signal Meaning
model.joblib exists in MODEL_OUTPUT_DIR Training artifact was produced successfully
FAST_CHECK_RESULT=PASS Fast smoke check completed successfully
FAST_CHECK_RESULT=WARN The check completed with a tolerated mismatch or warning path
FAST_CHECK_RESULT=FAIL The check failed
Prediction smoke test failed The smoke prediction raised an error
completed with WARNING The helper kept running after a tolerated mismatch
CustomJob resource name The training job was submitted
Endpoint resource name The model was deployed
Prediction response The live prediction path responded

Cleanup

[!NOTE] The verification guide distinguishes between a serving-contract mismatch and a transport failure. When the endpoint returns 'dict' object has no attribute 'predict', the verifier treats it as a contract mismatch, not a networking issue.

The guide includes a cleanup step after validation:

gcloud ai endpoints list --region "$REGION"
gcloud ai endpoints delete ENDPOINT_ID --region "$REGION"

This is used to remove no-longer-needed endpoints after verification to avoid continued serving cost.

Retry Semantics

The retry logic is the main operator safeguard when training or serving machine sizing is wrong for the workload.

Training retries

The tests in backend/tests/usecase/test_vertex_pipeline_machine_retry.py show that run_training_job retries across TRAIN_MACHINE_TYPES when the job fails with memory-related signals such as:

  • Replicas low on memory
  • code: 8
  • Resource exhausted

The same test file also verifies that non-memory failures are not retried and are surfaced directly.

Deployment retries

deploy_model retries across DEPLOY_MACHINE_TYPES when the serving container runs out of memory or the deployment path reports a quota problem. The tests verify both of these cases:

  • Model server container out of memory
  • 429 The following quotas are exceeded: CustomModelServingN2CPUsPerProjectPerRegion

When a deployment error is not retryable, deploy_model raises with a message that includes the failed machine_type and the original error text.

Error augmentation

The module includes an error-augmentation path for low-level trainer failures. The test coverage shows that a binary I/O failure in sklearn artifacts is rewritten into an actionable message containing:

  • filesystem read error
  • a pointer to build_and_push_training_image.sh

That makes corrupted or mismatched image layers easier to diagnose from job logs.

Retry flow

sequenceDiagram
    participant Ops as Operator
    participant Pipeline as run_vertex_pipeline module
    participant Job as Custom Job or Deploy call
    participant RetryList as Machine type list
    participant Vertex as Vertex AI

    Ops->>Pipeline: start training or deploy
    Pipeline->>RetryList: take first machine type
    Pipeline->>Job: run with current machine type
    Job-->>Pipeline: memory or quota error
    Pipeline->>RetryList: move to next machine type
    Pipeline->>Job: retry with next machine type
    Job-->>Pipeline: success
    Pipeline-->>Ops: endpoint or job resource name

Job state codes

The verification guide records these Custom Job state codes in logs:

Code Meaning
1 queued
2 pending or provisioning
3 running
4 succeeded
5 failed

Module Behavior Validated by Tests

backend/tests/usecase/test_vertex_pipeline_machine_retry.py

The test file confirms the following operational behaviors:

Scenario Verified behavior
Training memory failure Retries through the full TRAIN_MACHINE_TYPES list
Training non-memory failure Raises the original error
Deployment memory failure Retries through the full DEPLOY_MACHINE_TYPES list
Deployment quota failure Retries through the full DEPLOY_MACHINE_TYPES list
Deployment non-memory failure Raises a RuntimeError that names the failed machine type
Predict-only mode Uses existing model and endpoint resolution without training, upload, or deploy
Predict contract mismatch Continues when FAIL_ON_PREDICT_ERROR is disabled
Predict hard fail Raises when FAIL_ON_PREDICT_ERROR is enabled
Dict payload rejection Retries with a numeric vector
Feature-count mismatch Resizes the numeric vector to the expected length
Known feature count parsing Extracts 600 from sklearn mismatch messages
Numeric conversion Filters non-numeric values and keeps numeric booleans and numbers

Integration Points

  • backend/src/usecase/credit_score_runtime.py uses the deployed Vertex endpoint when the runtime provider is set to Vertex.
  • documents/vertex_ai_verification.md is the operator-facing reference for training, upload, deploy, predict, and cleanup.
  • platform/deployment/local/.env.example and platform/deployment/prod/.env.example are the deployment-level config entry points.
  • scripts/vertex_ai/* are the operational wrappers used to build, push, verify, and run the pipeline.

Observability

This section’s observability surface is intentionally log-driven and resource-driven:

  • the pipeline logs resource names for the Custom Job, uploaded model, endpoint, and prediction response,
  • the fast check writes a machine-friendly result line,
  • the report path is exported through FAST_CHECK_REPORT_PATH,
  • the verification guide maps endpoint readiness to traffic split and healthy replica status,
  • the operator is expected to inspect Vertex AI Custom Jobs and Endpoints in the Cloud Console when a run does not complete cleanly.

Error Handling

The runtime behavior is explicit in the tests and guide:

  • memory-related training and deployment failures are retried through the configured machine lists,
  • non-memory errors are surfaced immediately,
  • endpoint prediction shape mismatches can be treated as hard failures or tolerated warnings depending on VERIFY_ALLOW_SHAPE_MISMATCH,
  • prediction smoke failures can be fatal or non-fatal depending on FAIL_ON_PREDICT_ERROR,
  • resolve_existing_endpoint requires EXISTING_ENDPOINT_NAME and raises when it is missing.
Example operator outcome:
- prediction failed on deployed endpoint
- FAIL_ON_PREDICT_ERROR=0
- pipeline continues after logging the smoke failure
- report line still records the final status

Testing Considerations

The current coverage in backend/tests/usecase/test_vertex_pipeline_machine_retry.py is focused on rollout safety. The highest-value checks are:

  • memory retry paths for both training and deployment,
  • quota retry path for deployment,
  • non-retryable failure surfacing,
  • predict-only mode correctness,
  • shape mismatch detection and resize behavior,
  • numeric conversion behavior for mixed payloads,
  • error augmentation for low-level binary I/O failures.

Key Classes Reference

Class Responsibility
run_vertex_pipeline.py Orchestrates training image execution, model upload, endpoint deployment, prediction smoke tests, and machine-type retries
vertex_ai_verification.md Operator guide for build, push, deploy, verification, and cleanup
test_vertex_pipeline_machine_retry.py Verifies retry policy, endpoint prediction fallback, and failure classification
ensure_registry_and_auth.sh Prepares Artifact Registry auth and repository access for image pushes
build_and_push_training_image.sh Builds and pushes the training image with region-normalized registry hostnames
run_pipeline_via_compose.sh Runs the full Vertex pipeline through Compose
fast_credit_pipeline_check.sh Runs the repeatable smoke check for image import and runtime verification
verify_runtime_credit_scoring.sh Verifies a live deployed endpoint with SDK or gcloud fallback