Deployment, Operations, and Observability - ML Artifact Build, Vertex AI Deployment Verification, and Machine Retry Operations
Overview
This section covers the operator-facing flow for building the Vertex AI training image, pushing it to Artifact Registry, running the end-to-end pipeline, and verifying the deployed model before and after rollout. It is the fast reference for confirming that the training artifact, deployment target, and runtime prediction contract still line up after an image change or machine-type change.
The workflow is driven by deployment-level .env files and the helper scripts under scripts/vertex_ai/. The verification path is designed to work from the repo root through Docker Compose, while the retry policy inside run_vertex_pipeline.py automatically walks ordered machine-type lists when training or serving fails for memory-related reasons.
Architecture Overview
flowchart TB
subgraph Config[Deployment Configuration]
LocalEnv[platform deployment local env example]
ProdEnv[platform deployment prod env example]
end
subgraph Auth[Registry Auth and Access]
EnsureAuth[ensure_registry_and_auth sh]
DockerAuth[gcloud docker configure]
ArtifactRepo[Artifact Registry repo]
end
subgraph Build[Training Image Build]
BuildPush[build_and_push_training_image sh]
ComposeBuild[docker compose build]
NormalizedUri[Normalized TRAIN_IMAGE_URI]
end
subgraph Orchestration[Vertex Orchestration]
RunCompose[run_pipeline_via_compose sh]
FastCheck[fast_credit_pipeline_check sh]
VerifyRuntime[verify_runtime_credit_scoring sh]
PipelineModule[run_vertex_pipeline module]
end
subgraph VertexAI[Vertex AI]
CustomJob[Custom Job]
ModelUpload[Model upload]
EndpointDeploy[Endpoint deploy]
PredictCall[Endpoint predict]
end
subgraph Retry[Retry Policy]
TrainMachines[TRAIN_MACHINE_TYPES]
DeployMachines[DEPLOY_MACHINE_TYPES]
MemoryAndQuota[Memory and quota errors]
end
LocalEnv --> EnsureAuth
ProdEnv --> EnsureAuth
EnsureAuth --> DockerAuth
EnsureAuth --> ArtifactRepo
BuildPush --> EnsureAuth
BuildPush --> ComposeBuild
ComposeBuild --> NormalizedUri
RunCompose --> PipelineModule
RunCompose --> CustomJob
RunCompose --> ModelUpload
RunCompose --> EndpointDeploy
RunCompose --> PredictCall
FastCheck --> VerifyRuntime
VerifyRuntime --> PredictCall
PipelineModule --> CustomJob
PipelineModule --> ModelUpload
PipelineModule --> EndpointDeploy
PipelineModule --> PredictCall
TrainMachines --> CustomJob
DeployMachines --> EndpointDeploy
MemoryAndQuota --> TrainMachines
MemoryAndQuota --> DeployMachines
Deployment Configuration
Deployment level env examples
platform/deployment/local/.env.example
platform/deployment/prod/.env.example
These starter files are referenced by the verification guide and are the primary place operators are expected to set Vertex AI training and deployment values.
| Variable | Type | Purpose |
|---|---|---|
PROJECT_ID |
string | GCP project that owns the Artifact Registry repo and Vertex AI resources |
REGION |
string | Vertex AI region, used to normalize Artifact Registry hostnames |
TRAIN_IMAGE_URI |
string | Training image tag used by Compose and Vertex AI Custom Jobs |
DATA_URI |
string | Training data source passed to the trainer |
MODEL_OUTPUT_DIR |
string | GCS path where model.joblib is written |
PREDICTION_CONTAINER_IMAGE |
string | Prebuilt Vertex prediction container used for model serving |
GCP_CREDENTIALS_PATH |
string | Path to the service account key used by the registry helper |
SERVICE_ACCOUNT_EMAIL |
string | Service account used for impersonation when a key is not supplied |
DEPLOY_MACHINE_TYPE |
string | Single-machine serving shape override |
TRAIN_MACHINE_TYPE |
string | Primary training machine type |
TRAIN_MACHINE_TYPE_FALLBACK |
string | Fallback training machine type |
TRAIN_MACHINE_TYPES |
string list | Ordered training retry list |
DEPLOY_MACHINE_TYPES |
string list | Ordered serving retry list |
IMAGE_IMPORT_CHECK |
boolean | Enables the preflight Docker import check |
BUILD_BEFORE_RUN |
boolean | Controls whether run_pipeline_via_compose.sh rebuilds first |
PUSH_IMAGE_BEFORE_RUN |
boolean | Controls whether the freshly built image is pushed before the pipeline run |
RUN_TRAIN_STEP |
boolean | Enables the training stage in run_vertex_pipeline.py |
RUN_UPLOAD_STEP |
boolean | Enables the model upload stage |
RUN_DEPLOY_STEP |
boolean | Enables endpoint deployment |
RUN_PREDICT_STEP |
boolean | Enables the prediction smoke test |
FAIL_ON_PREDICT_ERROR |
boolean | Converts predict smoke failures into hard failures when enabled |
FAST_CHECK_REPORT_PATH |
string | Output path for the machine-friendly fast-check report |
VERIFY_STRICT |
boolean | Makes runtime verification fail when traffic is not ready |
WAIT_FOR_TRAFFIC_SPLIT_SECONDS |
integer | Wait window for endpoint traffic readiness |
VERIFY_ALLOW_SHAPE_MISMATCH |
boolean | Allows known serving-contract mismatch warnings to pass |
AUTO_SELECT_DEPLOYED_ENDPOINT |
boolean | Auto-selects a recent endpoint with active traffic |
EXISTING_ENDPOINT_NAME |
string | Used when running predict-only mode against an already deployed endpoint |
CREDIT_SCORE_PROVIDER |
string | Runtime provider selector for deployed Vertex scoring |
VERTEX_CREDIT_ENDPOINT_NAME |
string | Runtime endpoint name for Vertex credit scoring |
VERTEX_CREDIT_ENDPOINT_ID |
string | Runtime endpoint id for Vertex credit scoring |
CREDIT_SCORE_ALLOW_HEURISTIC_FALLBACK |
boolean | Controls fail-closed behavior in runtime scoring |
Command Entry Points
scripts/vertex_ai/ensure_registry_and_auth.sh
scripts/vertex_ai/build_and_push_training_image.sh
scripts/vertex_ai/run_pipeline_via_compose.sh
scripts/vertex_ai/fast_credit_pipeline_check.sh
scripts/vertex_ai/verify_runtime_credit_scoring.sh
| Script | Responsibility |
|---|---|
ensure_registry_and_auth.sh |
Activates registry credentials, configures Docker auth for Artifact Registry, creates the repo if missing, and grants writer access to the service account |
build_and_push_training_image.sh |
Wraps registry setup, normalizes the training image host to the active region, builds the training image, and pushes it |
run_pipeline_via_compose.sh |
Runs the training, upload, deploy, and smoke-predict pipeline through Compose |
fast_credit_pipeline_check.sh |
Performs a quick training-image import check and then runs runtime verification |
verify_runtime_credit_scoring.sh |
Verifies the live endpoint by running a prediction through the deployed credit-scoring path |
Training image push prerequisites
Before pushing the training image, the guide requires:
PROJECT_IDset in the deployment env file.gcloud auth logingcloud auth application-default logingcloud config set project "$PROJECT_ID"gcloud auth configure-docker us-central1-docker.pkg.dev- Either:
- a service account key at
./service-key.json, or - impersonation of
SERVICE_ACCOUNT_EMAIL
- a service account key at
The registry helper also creates the Artifact Registry repository if it does not exist and applies the writer grant needed for pushes to succeed.
Build platform requirement
[!IMPORTANT] The helpers normalize
TRAIN_IMAGE_URIso the hostname matchesREGION. This avoids pull failures when the env file still points at an older region host.
The training image build is forced to linux/amd64 during Compose build so the pushed image matches the platform expected by Vertex AI workers. The verification guide explicitly calls out this requirement to prevent manifest platform mismatches on Apple Silicon and other non-amd64 hosts.
Vertex Pipeline Orchestration Module
backend/src/credit-score-pipeline/run_vertex_pipeline.py
This module is the execution core behind the Vertex AI flow. The tests in backend/tests/usecase/test_vertex_pipeline_machine_retry.py show that it supports stage flags, machine-type retries, existing-endpoint predict-only runs, and shape-aware prediction retries.
Module properties
| Property | Type | Description |
|---|---|---|
TRAIN_MACHINE_TYPES |
list[str] | Ordered machine-type candidates for training retries |
DEPLOY_MACHINE_TYPES |
list[str] | Ordered machine-type candidates for deployment retries |
RUN_TRAIN_STEP |
bool | Enables or disables the training stage |
RUN_UPLOAD_STEP |
bool | Enables or disables the model upload stage |
RUN_DEPLOY_STEP |
bool | Enables or disables the deployment stage |
RUN_PREDICT_STEP |
bool | Enables or disables the prediction smoke test |
FAIL_ON_PREDICT_ERROR |
bool | Controls whether predict smoke failures abort the run |
EXISTING_ENDPOINT_NAME |
str or None | Required for predict-only verification against an existing endpoint |
Public methods
| Method | Description |
|---|---|
run_training_job |
Creates and runs the Vertex AI Custom Job, retrying across training machine types when memory-related errors occur |
upload_model |
Uploads the trained artifact for deployment |
deploy_model |
Creates an endpoint and deploys the model, retrying across serving machine types when memory or quota errors occur |
predict_example |
Runs the smoke prediction against the deployed endpoint and retries shape-sensitive payloads when needed |
resolve_existing_model |
Resolves the model for predict-only operation |
resolve_existing_endpoint |
Resolves the endpoint for predict-only operation |
main |
Executes the configured pipeline stages in order |
_augment_training_error |
Rewrites low-level trainer failures into operator-readable errors |
_is_memory_error |
Detects memory-constrained training and serving failures |
_is_instance_type_error |
Detects dict payload rejection during prediction |
_is_feature_count_mismatch_error |
Detects feature-shape mismatch failures |
_expected_feature_count |
Extracts the expected feature count from a mismatch error |
_resize_numeric_instance |
Pads or truncates numeric vectors to a target length |
_to_numeric_instance |
Converts mixed prediction inputs to a numeric list |
Stage initialization flow
- The module reads the stage flags and machine-type lists from env-backed constants.
- If
RUN_TRAIN_STEPis enabled, it callsrun_training_job. - If
RUN_UPLOAD_STEPis enabled, it uploads the resulting model. - If
RUN_DEPLOY_STEPis enabled, it deploys the model to Vertex AI. - If
RUN_PREDICT_STEPis enabled, it runs a smoke prediction. - If the predict step fails and
FAIL_ON_PREDICT_ERRORisfalse, the pipeline logs the failure and continues.
Data fetching and runtime verification flow
The predict path is intentionally contract-aware:
- it starts with a dictionary-style instance,
- retries with a numeric vector when the endpoint rejects the dict payload,
- retries again with a resized numeric vector when the endpoint reports a feature-count mismatch.
That behavior is verified by the unit tests that assert a three-step progression from dict payload to numeric list to resized numeric list.
Training Image Build and Registry Auth
build_and_push_training_image.sh
This helper is the entry point for image publishing. It wraps registry setup, normalizes the image URI host to the active region, and ensures the image that Vertex pulls is the same image Compose just built.
ensure_registry_and_auth.sh
This helper handles the registry access prerequisites for pushes:
- uses the service key at
GCP_CREDENTIALS_PATHwhen present, - otherwise impersonates
SERVICE_ACCOUNT_EMAIL, - configures Docker for the Artifact Registry host derived from
TRAIN_IMAGE_URI, - creates the Artifact Registry repo if it is missing,
- grants writer access to the service account so pushes succeed.
Build and push flow
sequenceDiagram
participant Ops as Operator
participant Build as build_and_push_training_image sh
participant Auth as ensure_registry_and_auth sh
participant Gcloud as gcloud
participant Docker as Docker
participant Repo as Artifact Registry
Ops->>Build: run with deployment env file
Build->>Auth: prepare registry access
Auth->>Gcloud: activate key or impersonate service account
Auth->>Gcloud: configure docker auth
Auth->>Repo: create repo if missing
Auth->>Repo: grant writer access
Build->>Docker: compose build
Build->>Docker: compose push
Build-->>Ops: normalized image pushed
Vertex AI Deployment Verification
The verification guide documents two related checks:
- a full orchestration run that trains, uploads, deploys, and runs a smoke prediction, and
- a fast repeatable smoke check that verifies the image import and endpoint runtime contract without retraining.
Verification semantics
Full pipeline run
run_pipeline_via_compose.sh is the operator path for the end-to-end job. It is documented to:
- build the
vertex-ai-pipelineservice before running by default, - push the same image before the Custom Job starts,
- submit the Custom Job,
- upload the artifact,
- deploy the endpoint,
- run a test prediction,
- log the CustomJob, model upload, endpoint, and prediction resource results.
Fast smoke check
fast_credit_pipeline_check.sh performs two checks:
- a small sklearn import check using
ExtraTreesClassifierto catch corrupted training images early, - runtime verification of the deployed endpoint.
The helper supports:
IMAGE_IMPORT_CHECK=0to skip the image import check,VERIFY_STRICT=1to fail when no endpoint traffic is ready,WAIT_FOR_TRAFFIC_SPLIT_SECONDSto tune how long it waits,VERIFY_ALLOW_SHAPE_MISMATCH=0to fail hard on serving-contract mismatches,VERIFY_ALLOW_SHAPE_MISMATCH=1to tolerate known legacy artifact issues with a warning.
When the warning path is used, the helper emits an explicit completed with WARNING summary and writes a final machine-readable status line.
Runtime endpoint verification
verify_runtime_credit_scoring.sh is the live endpoint verifier. The guide says it:
- prefers the Python SDK
google-cloud-aiplatform, - falls back to
gcloud ai endpoints predictwhen the SDK is missing, - auto-selects a recently updated endpoint with active traffic when the configured endpoint has no traffic split yet,
- waits briefly in non-strict mode,
- waits longer and fails in strict mode when traffic never becomes ready.
Verification flow
sequenceDiagram
participant Ops as Operator
participant Check as fast_credit_pipeline_check sh
participant Runtime as verify_runtime_credit_scoring sh
participant SDK as Vertex SDK or gcloud
participant Endpoint as Deployed Endpoint
Ops->>Check: run fast smoke check
Check->>Check: import training image
Check->>Runtime: verify live endpoint
Runtime->>SDK: predict request
SDK->>Endpoint: endpoint predict
Endpoint-->>SDK: predictions
SDK-->>Runtime: result
Runtime-->>Check: pass warn or fail
Check-->>Ops: FAST_CHECK_RESULT
Success and failure signals
| Signal | Meaning |
|---|---|
model.joblib exists in MODEL_OUTPUT_DIR |
Training artifact was produced successfully |
FAST_CHECK_RESULT=PASS |
Fast smoke check completed successfully |
FAST_CHECK_RESULT=WARN |
The check completed with a tolerated mismatch or warning path |
FAST_CHECK_RESULT=FAIL |
The check failed |
Prediction smoke test failed |
The smoke prediction raised an error |
completed with WARNING |
The helper kept running after a tolerated mismatch |
| CustomJob resource name | The training job was submitted |
| Endpoint resource name | The model was deployed |
| Prediction response | The live prediction path responded |
Cleanup
[!NOTE] The verification guide distinguishes between a serving-contract mismatch and a transport failure. When the endpoint returns
'dict' object has no attribute 'predict', the verifier treats it as a contract mismatch, not a networking issue.
The guide includes a cleanup step after validation:
gcloud ai endpoints list --region "$REGION"
gcloud ai endpoints delete ENDPOINT_ID --region "$REGION"
This is used to remove no-longer-needed endpoints after verification to avoid continued serving cost.
Retry Semantics
The retry logic is the main operator safeguard when training or serving machine sizing is wrong for the workload.
Training retries
The tests in backend/tests/usecase/test_vertex_pipeline_machine_retry.py show that run_training_job retries across TRAIN_MACHINE_TYPES when the job fails with memory-related signals such as:
Replicas low on memorycode: 8Resource exhausted
The same test file also verifies that non-memory failures are not retried and are surfaced directly.
Deployment retries
deploy_model retries across DEPLOY_MACHINE_TYPES when the serving container runs out of memory or the deployment path reports a quota problem. The tests verify both of these cases:
Model server container out of memory429 The following quotas are exceeded: CustomModelServingN2CPUsPerProjectPerRegion
When a deployment error is not retryable, deploy_model raises with a message that includes the failed machine_type and the original error text.
Error augmentation
The module includes an error-augmentation path for low-level trainer failures. The test coverage shows that a binary I/O failure in sklearn artifacts is rewritten into an actionable message containing:
filesystem read error- a pointer to
build_and_push_training_image.sh
That makes corrupted or mismatched image layers easier to diagnose from job logs.
Retry flow
sequenceDiagram
participant Ops as Operator
participant Pipeline as run_vertex_pipeline module
participant Job as Custom Job or Deploy call
participant RetryList as Machine type list
participant Vertex as Vertex AI
Ops->>Pipeline: start training or deploy
Pipeline->>RetryList: take first machine type
Pipeline->>Job: run with current machine type
Job-->>Pipeline: memory or quota error
Pipeline->>RetryList: move to next machine type
Pipeline->>Job: retry with next machine type
Job-->>Pipeline: success
Pipeline-->>Ops: endpoint or job resource name
Job state codes
The verification guide records these Custom Job state codes in logs:
| Code | Meaning |
|---|---|
1 |
queued |
2 |
pending or provisioning |
3 |
running |
4 |
succeeded |
5 |
failed |
Module Behavior Validated by Tests
backend/tests/usecase/test_vertex_pipeline_machine_retry.py
The test file confirms the following operational behaviors:
| Scenario | Verified behavior |
|---|---|
| Training memory failure | Retries through the full TRAIN_MACHINE_TYPES list |
| Training non-memory failure | Raises the original error |
| Deployment memory failure | Retries through the full DEPLOY_MACHINE_TYPES list |
| Deployment quota failure | Retries through the full DEPLOY_MACHINE_TYPES list |
| Deployment non-memory failure | Raises a RuntimeError that names the failed machine type |
| Predict-only mode | Uses existing model and endpoint resolution without training, upload, or deploy |
| Predict contract mismatch | Continues when FAIL_ON_PREDICT_ERROR is disabled |
| Predict hard fail | Raises when FAIL_ON_PREDICT_ERROR is enabled |
| Dict payload rejection | Retries with a numeric vector |
| Feature-count mismatch | Resizes the numeric vector to the expected length |
| Known feature count parsing | Extracts 600 from sklearn mismatch messages |
| Numeric conversion | Filters non-numeric values and keeps numeric booleans and numbers |
Integration Points
backend/src/usecase/credit_score_runtime.pyuses the deployed Vertex endpoint when the runtime provider is set to Vertex.documents/vertex_ai_verification.mdis the operator-facing reference for training, upload, deploy, predict, and cleanup.platform/deployment/local/.env.exampleandplatform/deployment/prod/.env.exampleare the deployment-level config entry points.scripts/vertex_ai/*are the operational wrappers used to build, push, verify, and run the pipeline.
Observability
This section’s observability surface is intentionally log-driven and resource-driven:
- the pipeline logs resource names for the Custom Job, uploaded model, endpoint, and prediction response,
- the fast check writes a machine-friendly result line,
- the report path is exported through
FAST_CHECK_REPORT_PATH, - the verification guide maps endpoint readiness to traffic split and healthy replica status,
- the operator is expected to inspect Vertex AI Custom Jobs and Endpoints in the Cloud Console when a run does not complete cleanly.
Error Handling
The runtime behavior is explicit in the tests and guide:
- memory-related training and deployment failures are retried through the configured machine lists,
- non-memory errors are surfaced immediately,
- endpoint prediction shape mismatches can be treated as hard failures or tolerated warnings depending on
VERIFY_ALLOW_SHAPE_MISMATCH, - prediction smoke failures can be fatal or non-fatal depending on
FAIL_ON_PREDICT_ERROR, resolve_existing_endpointrequiresEXISTING_ENDPOINT_NAMEand raises when it is missing.
Example operator outcome:
- prediction failed on deployed endpoint
- FAIL_ON_PREDICT_ERROR=0
- pipeline continues after logging the smoke failure
- report line still records the final status
Testing Considerations
The current coverage in backend/tests/usecase/test_vertex_pipeline_machine_retry.py is focused on rollout safety. The highest-value checks are:
- memory retry paths for both training and deployment,
- quota retry path for deployment,
- non-retryable failure surfacing,
- predict-only mode correctness,
- shape mismatch detection and resize behavior,
- numeric conversion behavior for mixed payloads,
- error augmentation for low-level binary I/O failures.
Key Classes Reference
| Class | Responsibility |
|---|---|
run_vertex_pipeline.py |
Orchestrates training image execution, model upload, endpoint deployment, prediction smoke tests, and machine-type retries |
vertex_ai_verification.md |
Operator guide for build, push, deploy, verification, and cleanup |
test_vertex_pipeline_machine_retry.py |
Verifies retry policy, endpoint prediction fallback, and failure classification |
ensure_registry_and_auth.sh |
Prepares Artifact Registry auth and repository access for image pushes |
build_and_push_training_image.sh |
Builds and pushes the training image with region-normalized registry hostnames |
run_pipeline_via_compose.sh |
Runs the full Vertex pipeline through Compose |
fast_credit_pipeline_check.sh |
Runs the repeatable smoke check for image import and runtime verification |
verify_runtime_credit_scoring.sh |
Verifies a live deployed endpoint with SDK or gcloud fallback |