Credit Scoring and Pipeline DOMAIN - Vertex AI Training, Deployment Workflow, and Verification Runbooks

Overview

This section documents the Vertex AI operational path used to train, register, deploy, and verify the credit-scoring model used by Lita’s housing workflow. The repository treats the ML lifecycle as a Compose-driven pipeline: local or prod deployment env files supply the configuration, a training container is built and pushed to Artifact Registry, Vertex AI Custom Jobs run the trainer, the resulting artifact is uploaded as a model, and an endpoint is deployed and smoke-tested.

The same operational contract is reused for verification. The runbooks in documents/vertex_ai_verification.md and the helpers under scripts/vertex_ai/* provide repeatable checks for registry auth, image tagging, machine-type retries, model upload/deploy readiness, and runtime endpoint validation. The runtime provider in backend/src/usecase/credit_score_runtime.py consumes the same Vertex endpoint configuration at application time, so the deployment env files participate in both the training lifecycle and the live scoring lifecycle.

Architecture Overview

flowchart TB
    subgraph ConfigFiles [Deployment Configuration]
        LocalEnv[platform/deployment/local/.env.example]
        ProdEnv[platform/deployment/prod/.env.example]
        DeployEnv[platform/deployment/local/.env or prod/.env]
    end

    subgraph Scripts [Vertex AI Runbooks]
        EnsureAuth[scripts/vertex_ai/ensure_registry_and_auth.sh]
        BuildPush[scripts/vertex_ai/build_and_push_training_image.sh]
        RunPipeline[scripts/vertex_ai/run_pipeline_via_compose.sh]
        FastCheck[scripts/vertex_ai/fast_credit_pipeline_check.sh]
        VerifyRuntime[scripts/vertex_ai/verify_runtime_credit_scoring.sh]
    end

    subgraph ComposePipeline [Compose Orchestrated Pipeline]
        TrainJob[run_vertex_pipeline.py train step]
        UploadModel[run_vertex_pipeline.py upload step]
        DeployEndpoint[run_vertex_pipeline.py deploy step]
        SmokePredict[run_vertex_pipeline.py predict step]
    end

    subgraph VertexAI [Vertex AI]
        CustomJob[Custom Job]
        ModelRegistry[Model Registry]
        Endpoint[Deployed Endpoint]
    end

    subgraph Runtime [Application Runtime]
        CreditRuntime[backend/src/usecase/credit_score_runtime.py]
    end

    LocalEnv --> DeployEnv
    ProdEnv --> DeployEnv

    DeployEnv --> EnsureAuth
    DeployEnv --> BuildPush
    DeployEnv --> RunPipeline
    DeployEnv --> FastCheck
    DeployEnv --> VerifyRuntime

    EnsureAuth --> BuildPush
    BuildPush --> RunPipeline
    RunPipeline --> TrainJob
    TrainJob --> CustomJob
    TrainJob --> UploadModel
    UploadModel --> ModelRegistry
    UploadModel --> DeployEndpoint
    DeployEndpoint --> Endpoint
    DeployEndpoint --> SmokePredict
    SmokePredict --> Endpoint
    VerifyRuntime --> Endpoint
    CreditRuntime --> Endpoint

Operational Components

1. Deployment Configuration

Deployment Env Files

[!IMPORTANT] The pipeline is controlled by deployment-level env files, not by ad hoc local state. The same values for PROJECT_ID, REGION, TRAIN_IMAGE_URI, DATA_URI, MODEL_OUTPUT_DIR, and the Vertex endpoint identifiers are used across build, train, deploy, and runtime verification steps.

platform/deployment/local/.env.example, platform/deployment/prod/.env.example, platform/deployment/local/.env, platform/deployment/prod/.env

These files drive the ML lifecycle from the Compose layer. The verification guide uses them to keep build, push, train, deploy, and runtime validation aligned with the same project, region, and registry settings.

Variable Role in the Vertex AI lifecycle
PROJECT_ID Used to derive Artifact Registry paths and Vertex resource names
REGION Used for Vertex AI and Artifact Registry host normalization
DATA_URI Training input CSV for the trainer entrypoint
MODEL_OUTPUT_DIR GCS path where model.joblib is written
TRAIN_IMAGE_URI Training container image tag used by Custom Jobs
PREDICTION_CONTAINER_IMAGE Serving container image used during model import/deploy
VERTEX_CREDIT_ENDPOINT_NAME Preferred endpoint resource name for runtime scoring
VERTEX_CREDIT_ENDPOINT_ID Alternate endpoint selector for runtime scoring
CREDIT_SCORE_PROVIDER Must be set to vertex for Vertex-based runtime scoring
CREDIT_SCORE_ALLOW_HEURISTIC_FALLBACK Fail-closed toggle for runtime scoring
SERVICE_ACCOUNT_EMAIL Used by auth helpers when impersonation is needed
GCP_CREDENTIALS_PATH Path to the service account key used by registry helpers

2. Training Image Preparation

Training Image Build and Push Helper

scripts/vertex_ai/build_and_push_training_image.sh

This helper prepares the Docker image that Vertex AI Custom Jobs execute. The verification guide shows that it wraps registry/auth setup, normalizes TRAIN_IMAGE_URI to match REGION, and then runs Compose build/push so the image tag used by the training job always matches the image that was actually pushed.

Step Behavior
Registry setup Calls scripts/vertex_ai/ensure_registry_and_auth.sh first
Auth setup Uses service-key.json by default when present, or impersonates the configured service account
Image normalization Forces the registry host to match the configured REGION
Architecture Forces linux/amd64 during build so Vertex workers can pull the image
Output Pushes us-central1-docker.pkg.dev/${PROJECT_ID}/transunion/pipeline:latest unless overridden

Registry and Auth Helper

scripts/vertex_ai/ensure_registry_and_auth.sh

This helper prepares Artifact Registry access before the build/push step. The verification guide states that it configures Docker for the registry host derived from TRAIN_IMAGE_URI, creates the repository if missing, and grants writer access to the selected service account.

3. Vertex AI Pipeline Orchestrator

Vertex AI Pipeline Runner

backend/src/credit-score-pipeline/run_vertex_pipeline.py

This module is the operational center of the Vertex AI workflow. The tests in backend/tests/usecase/test_vertex_pipeline_machine_retry.py verify the behaviors that matter for deployment resilience: machine-type retry, error augmentation, predict-only execution, and endpoint-based runtime checks.

Method Description
run_training_job Submits the Custom Job and retries across TRAIN_MACHINE_TYPES until a memory-safe machine succeeds
upload_model Uploads the trained artifact to Vertex AI for registration
deploy_model Deploys the uploaded model to an endpoint and retries across DEPLOY_MACHINE_TYPES
predict_example Sends a smoke prediction to the deployed endpoint and retries when the serving contract rejects the first payload
resolve_existing_model Resolves an already-registered model when the run is predict-only or skips retraining
resolve_existing_endpoint Resolves an already-deployed endpoint for runtime verification
main Orchestrates the enabled pipeline steps controlled by the run flags
_is_memory_error Detects memory-related job failures
_augment_training_error Adds actionable context to training failures
_is_instance_type_error Detects dict-to-float serving-contract errors
_is_feature_count_mismatch_error Detects sklearn feature-count mismatch errors
_expected_feature_count Extracts the expected input width from sklearn error text
_resize_numeric_instance Pads or trims numeric vectors to the expected width
_to_numeric_instance Filters mixed payloads down to numeric values

Trainer Entrypoint

backend/src/credit-score-pipeline/junya_credit_score_model.py

This is the entrypoint invoked by the Vertex training container. The verification guide shows the runtime command pattern:

python ../../backend/src/vertex_ai_pipeline/junya_credit_score_model.py \
  --data_uri "$DATA_URI" --model_output_dir "$MODEL_OUTPUT_DIR"

The documented container flow uses the Dockerfile ENTRYPOINT ["python", "junya_credit_score_model.py"], so the console path leaves the command empty and passes only arguments.

4. Runtime Endpoint Configuration

Vertex Runtime Provider

backend/src/usecase/credit_score_runtime.py

This module is the runtime bridge that sends the application’s feature payload to a Vertex endpoint. It resolves project and endpoint identity from env vars, flattens the feature payload into a sklearn-safe numeric shape, calls the Vertex endpoint, and validates the returned contract.

Method Description
_require_project_id Reads PROJECT_ID or GCP_PROJECT_ID and raises if neither is present
_resolve_endpoint_name Builds a full Vertex endpoint resource name from VERTEX_CREDIT_ENDPOINT_NAME or VERTEX_CREDIT_ENDPOINT_ID
_flatten_features_for_vertex Converts nested and mixed-type feature data into a flat numeric dictionary
_score_with_vertex Initializes Vertex AI and calls endpoint.predict
score_housing_application Returns the application’s credit-scoring result using the configured provider

Runtime Endpoint Resolution

The endpoint resolver accepts either a full resource name or a short ID.

Input Result
projects/.../locations/.../endpoints/... Used as-is
123456 Expanded to projects/{project_id}/locations/{region}/endpoints/123456

Runtime Contract Inputs

The verification guide requires these runtime variables:

  • CREDIT_SCORE_PROVIDER=vertex
  • PROJECT_ID or GCP_PROJECT_ID
  • REGION
  • VERTEX_CREDIT_ENDPOINT_NAME or VERTEX_CREDIT_ENDPOINT_ID

Optional fail-closed behavior is controlled by:

  • CREDIT_SCORE_ALLOW_HEURISTIC_FALLBACK=false

5. Verification and Smoke Test Helpers

Fast Pipeline Smoke Check

scripts/vertex_ai/fast_credit_pipeline_check.sh

This helper performs a quick PASS/FAIL validation after an endpoint already exists. The verification guide states that it:

  1. Pulls TRAIN_IMAGE_URI
  2. Runs a small training-image import check using ExtraTreesClassifier
  3. Runs runtime endpoint verification through verify_runtime_credit_scoring.sh

Useful toggles documented in the runbook:

Variable Effect
IMAGE_IMPORT_CHECK=0 Skips the training-image import sanity check
VERIFY_STRICT=1 Fails when endpoint traffic or predict readiness is missing
WAIT_FOR_TRAFFIC_SPLIT_SECONDS=300 Adjusts how long the helper waits for traffic split readiness
VERIFY_ALLOW_SHAPE_MISMATCH=0 Fails on known serving-contract mismatches
FAST_CHECK_REPORT_PATH Writes a final report file
FAIL_ON_PREDICT_ERROR Controls whether predict smoke failures are fatal

The guide also states that the helper ends with a machine-readable status line:

  • FAST_CHECK_RESULT=PASS
  • FAST_CHECK_RESULT=WARN
  • FAST_CHECK_RESULT=FAIL

Runtime Credit Scoring Verifier

scripts/vertex_ai/verify_runtime_credit_scoring.sh

This helper validates the live Vertex endpoint separately from retraining. The guide states that it prefers the Python SDK (google-cloud-aiplatform) and falls back to gcloud ai endpoints predict when the SDK is unavailable.

Behavior Details
Endpoint discovery Uses VERTEX_CREDIT_ENDPOINT_NAME or VERTEX_CREDIT_ENDPOINT_ID
Traffic handling Auto-selects a deployed endpoint with active traffic when needed
Strict mode Waits longer and fails when traffic never becomes ready
Predict shape handling Can fail hard or continue with warning for known contract mismatches

Training and Deployment Workflow

Build the Training Image

The training image is built and pushed before the pipeline run so Vertex Jobs consume the exact image version that was validated locally. The runbook explicitly says this prevents stale image usage and region mismatch issues.

cd "$REPO_ROOT"
scripts/vertex_ai/build_and_push_training_image.sh platform/deployment/local/.env

Key behaviors from the runbook:

  • It is safe to pass a different env file for prod.
  • If TRAIN_IMAGE_URI is omitted, the compose stack derives a tag using PROJECT_ID.
  • If the registry host in TRAIN_IMAGE_URI does not match REGION, the helper normalizes it.
  • The build is forced to linux/amd64 to match Vertex worker architecture.

Run the Vertex AI Pipeline

The orchestrator runs through Docker Compose so the training container and the orchestration environment stay in sync.

BUILD_BEFORE_RUN=1 scripts/vertex_ai/run_pipeline_via_compose.sh platform/deployment/local/.env

The guide documents three step gates:

  • RUN_TRAIN_STEP
  • RUN_UPLOAD_STEP
  • RUN_DEPLOY_STEP
  • RUN_PREDICT_STEP

It also documents the failure toggle:

  • FAIL_ON_PREDICT_ERROR

Operational output from the runner includes:

  • Custom Job resource name
  • Model upload resource name
  • Endpoint resource name
  • Prediction response

Register and Deploy in Vertex AI

The console-only path mirrors the automated workflow:

  1. Launch a Vertex AI Custom Training job.
  2. Confirm model.joblib exists in MODEL_OUTPUT_DIR.
  3. Import the model from GCS.
  4. Deploy an endpoint.
  5. Run a test prediction.

The runbook explicitly uses:

  • training image URI from TRAIN_IMAGE_URI
  • artifact URI like gs://transuniondataset/related_files_we_made/models/transunion_v1
  • serving container us-docker.pkg.dev/vertex-ai/prediction/sklearn-cpu.1-4:latest

Verify the Result

The verification doc describes the expected end state:

  • model.joblib exists in GCS under the configured model output directory
  • Vertex AI Custom Jobs shows the job as successful
  • Vertex AI Endpoints shows healthy replicas
  • The smoke prediction returns a predictions array

Retry and Error Handling

Training Job Retry Semantics

The tests in backend/tests/usecase/test_vertex_pipeline_machine_retry.py show that run_training_job retries machine types in order until the job succeeds.

Rule Behavior
Memory-related failure Retry on the next TRAIN_MACHINE_TYPES entry
Non-memory failure Re-raise the error
Binary I/O import failure _augment_training_error adds a filesystem-read hint and points to build_and_push_training_image.sh

The helper recognizes these memory-related patterns:

  • code: 8
  • Resource exhausted while scheduling

It also surfaces a known sklearn binary error as a filesystem-read problem so the operator can rebuild the training image.

Deployment Retry Semantics

deploy_model retries across DEPLOY_MACHINE_TYPES when deployment fails for known capacity-related reasons.

Failure pattern Observed behavior
Model server out of memory Retries with the next deployment machine type
Quota exhausted Retries with the next deployment machine type
Other runtime error Re-raises with machine-type context

The tests verify that the helper keeps advancing until a machine type succeeds.

Prediction Smoke Test Handling

The smoke prediction step is intentionally tolerant of contract mismatches when configured to be non-fatal.

Case Behavior
FAIL_ON_PREDICT_ERROR=0 Logs a skipped smoke test and continues
FAIL_ON_PREDICT_ERROR=1 Re-raises the prediction failure
float() argument must be a string or a real number, not 'dict' Detected as a Vertex serving-contract mismatch
Feature-count mismatch The helper can resize the numeric vector and retry

The tests show three retry shapes in the prediction path:

  1. First attempt with the raw dict payload
  2. Second attempt with a numeric vector
  3. Third attempt with a resized vector when a feature-count mismatch is detected

Known Error Patterns Covered by Tests

Helper Pattern it matches
_is_instance_type_error Wrapped dict to numeric conversion failures
_is_feature_count_mismatch_error sklearn feature-count mismatch strings and wrapped payloads
_expected_feature_count Extracts the expected input width from sklearn error text
_resize_numeric_instance Pads a numeric list to the expected feature width

Verification Runbooks

1. Quick End-to-End Smoke Check

[!NOTE] The tests show that a deploy-time or predict-time mismatch is treated as a serving-contract issue, not as a transport issue. The runtime helper distinguishes these cases so operational failures are reported with the right retry or skip behavior.

Run this when you only need a fast verification against an existing endpoint.

scripts/vertex_ai/fast_credit_pipeline_check.sh platform/deployment/local/.env

What this checks:

  • training image import sanity
  • endpoint readiness
  • runtime prediction path
  • final report output

2. Runtime-Only Verification

Run this when the model is already deployed and you only want to validate endpoint readiness.

VERIFY_STRICT=1 WAIT_FOR_TRAFFIC_SPLIT_SECONDS=600 \
scripts/vertex_ai/verify_runtime_credit_scoring.sh platform/deployment/local/.env

The guide says this:

  • waits for traffic split readiness
  • performs endpoint.predict
  • fails in strict mode if readiness never arrives

3. Full Pipeline Run

Run this when you need the entire train → upload → deploy → predict sequence.

BUILD_BEFORE_RUN=1 scripts/vertex_ai/run_pipeline_via_compose.sh platform/deployment/local/.env

The runbook also documents:

  • BUILD_BEFORE_RUN=0 to skip rebuilding
  • PUSH_IMAGE_BEFORE_RUN=0 to skip pushing
  • RUN_DEPLOY_STEP=0 to stop after model upload
  • RUN_PREDICT_STEP=0 to skip the smoke prediction

4. Console-Only Validation

Use the Vertex UI if you want to verify the path manually.

{
  "income_value": 120000.0,
  "savings_value": 50000.0,
  "budget_value": 400000.0,
  "coverage_score": 0.58,
  "days_covered": 92,
  "salary_detected": 1.0,
  "pd_hint": "endpoint should respond with pd_p50_12m/pd_p90_12m contract"
}

The runbook says that a predictions array confirms the end-to-end path is working.

5. Cleanup

After validation, delete the endpoint if it is no longer needed.

gcloud ai endpoints list --region "$REGION"
gcloud ai endpoints delete ENDPOINT_ID --region "$REGION"

Sequence Flows

Build, Train, Upload, Deploy, and Predict

sequenceDiagram
    participant Op as Operator
    participant Script as run_pipeline_via_compose.sh
    participant Compose as Docker Compose
    participant Runner as run_vertex_pipeline.py
    participant Job as Vertex Custom Job
    participant Reg as Model Registry
    participant Ep as Vertex Endpoint

    Op->>Script: start pipeline with deployment env file
    Script->>Compose: build and push training image
    Compose->>Runner: start orchestration with env settings
    Runner->>Job: submit training job
    Job-->>Runner: training artifact written to MODEL_OUTPUT_DIR
    Runner->>Reg: upload model artifact
    Reg-->>Runner: model resource name
    Runner->>Ep: deploy model to endpoint
    Ep-->>Runner: endpoint resource name
    Runner->>Ep: run smoke prediction
    Ep-->>Runner: predictions array or contract error
    Runner-->>Op: logs resource names and final status

Runtime Endpoint Verification

sequenceDiagram
    participant Op as Operator
    participant Verify as verify_runtime_credit_scoring.sh
    participant SDK as google-cloud-aiplatform or gcloud
    participant Ep as Vertex Endpoint
    participant App as credit_score_runtime.py

    Op->>Verify: run runtime verification with env file
    Verify->>SDK: initialize project and location
    Verify->>Ep: predict with flattened feature payload
    Ep-->>Verify: prediction response
    Verify-->>Op: readiness result and contract status
    App->>Ep: score_housing_application uses same endpoint settings

Training Retry and Fallback Flow

flowchart TD
    Start([run_training_job])
    TryMachine[Select machine type from TRAIN_MACHINE_TYPES]
    RunJob[Submit Custom Job]
    MemoryError{Memory related error}
    Success([Job succeeds])
    Augment[Augment error message]
    NextMachine[Try next machine type]
    Fail([Raise error])

    Start --> TryMachine
    TryMachine --> RunJob
    RunJob --> MemoryError
    MemoryError -->|yes| NextMachine
    NextMachine --> TryMachine
    MemoryError -->|no| Augment
    Augment --> Fail
    RunJob --> Success

Environment and Lifecycle Participation

Local and Prod Env Files

The verification guide explicitly uses deployment-level env files so the same pipeline can run against the local or prod Docker Compose stack.

File Role
platform/deployment/local/.env.example Starter file for local Vertex AI configuration
platform/deployment/prod/.env.example Starter file for production Vertex AI configuration
platform/deployment/local/.env Actual local run configuration
platform/deployment/prod/.env Actual production run configuration

Lifecycle Flow

  1. Populate the env file.
  2. Authenticate with Google Cloud.
  3. Configure Docker for Artifact Registry.
  4. Build and push the training image.
  5. Run the pipeline via Compose.
  6. Verify the endpoint.
  7. Clean up the endpoint when done.

Credential and Registry Participation

The verification guide supports both:

This lets the same helpers run in local and CI-like contexts without changing the pipeline contract.

Testing Considerations

The tests in backend/tests/usecase/test_vertex_pipeline_machine_retry.py encode the expected operational behavior for the pipeline.

Scenario What is verified
Memory-heavy training failure Retries to the next training machine type
Non-memory training error Propagates the error
Invalid endpoint env resolve_existing_endpoint requires EXISTING_ENDPOINT_NAME
Deployment quota exhaustion Retries to the next deployment machine type
Deployment OOM Retries to the next deployment machine type
Predict-only mode Uses the existing endpoint without retraining or redeploying
Fatal predict mode Re-raises prediction errors when FAIL_ON_PREDICT_ERROR=True
Non-fatal predict mode Logs a skipped smoke test and continues
Shape mismatch Resizes the numeric vector and retries
Wrapped dict-to-float error Recognized as an instance-type contract issue

Key Classes Reference

Class Responsibility
run_vertex_pipeline.py Orchestrates training, model upload, endpoint deployment, and smoke prediction with retry handling
junya_credit_score_model.py Training entrypoint executed inside the Vertex AI Custom Job container
credit_score_runtime.py Resolves the Vertex endpoint and scores applications at runtime
vertex_ai_verification.md Runbook for registry auth, image preparation, pipeline execution, and verification
test_vertex_pipeline_machine_retry.py Verifies retry, smoke-test, and contract-mismatch behavior
build_and_push_training_image.sh Builds and pushes the training image with normalized registry settings
run_pipeline_via_compose.sh Runs the full pipeline through Docker Compose
verify_runtime_credit_scoring.sh Validates the live Vertex endpoint contract
fast_credit_pipeline_check.sh Runs the fast image-and-endpoint smoke check