Credit Scoring and Pipeline DOMAIN - Vertex AI Training, Deployment Workflow, and Verification Runbooks
Overview
This section documents the Vertex AI operational path used to train, register, deploy, and verify the credit-scoring model used by Lita’s housing workflow. The repository treats the ML lifecycle as a Compose-driven pipeline: local or prod deployment env files supply the configuration, a training container is built and pushed to Artifact Registry, Vertex AI Custom Jobs run the trainer, the resulting artifact is uploaded as a model, and an endpoint is deployed and smoke-tested.
The same operational contract is reused for verification. The runbooks in documents/vertex_ai_verification.md and the helpers under scripts/vertex_ai/* provide repeatable checks for registry auth, image tagging, machine-type retries, model upload/deploy readiness, and runtime endpoint validation. The runtime provider in backend/src/usecase/credit_score_runtime.py consumes the same Vertex endpoint configuration at application time, so the deployment env files participate in both the training lifecycle and the live scoring lifecycle.
Architecture Overview
flowchart TB
subgraph ConfigFiles [Deployment Configuration]
LocalEnv[platform/deployment/local/.env.example]
ProdEnv[platform/deployment/prod/.env.example]
DeployEnv[platform/deployment/local/.env or prod/.env]
end
subgraph Scripts [Vertex AI Runbooks]
EnsureAuth[scripts/vertex_ai/ensure_registry_and_auth.sh]
BuildPush[scripts/vertex_ai/build_and_push_training_image.sh]
RunPipeline[scripts/vertex_ai/run_pipeline_via_compose.sh]
FastCheck[scripts/vertex_ai/fast_credit_pipeline_check.sh]
VerifyRuntime[scripts/vertex_ai/verify_runtime_credit_scoring.sh]
end
subgraph ComposePipeline [Compose Orchestrated Pipeline]
TrainJob[run_vertex_pipeline.py train step]
UploadModel[run_vertex_pipeline.py upload step]
DeployEndpoint[run_vertex_pipeline.py deploy step]
SmokePredict[run_vertex_pipeline.py predict step]
end
subgraph VertexAI [Vertex AI]
CustomJob[Custom Job]
ModelRegistry[Model Registry]
Endpoint[Deployed Endpoint]
end
subgraph Runtime [Application Runtime]
CreditRuntime[backend/src/usecase/credit_score_runtime.py]
end
LocalEnv --> DeployEnv
ProdEnv --> DeployEnv
DeployEnv --> EnsureAuth
DeployEnv --> BuildPush
DeployEnv --> RunPipeline
DeployEnv --> FastCheck
DeployEnv --> VerifyRuntime
EnsureAuth --> BuildPush
BuildPush --> RunPipeline
RunPipeline --> TrainJob
TrainJob --> CustomJob
TrainJob --> UploadModel
UploadModel --> ModelRegistry
UploadModel --> DeployEndpoint
DeployEndpoint --> Endpoint
DeployEndpoint --> SmokePredict
SmokePredict --> Endpoint
VerifyRuntime --> Endpoint
CreditRuntime --> Endpoint
Operational Components
1. Deployment Configuration
Deployment Env Files
[!IMPORTANT] The pipeline is controlled by deployment-level env files, not by ad hoc local state. The same values for
PROJECT_ID,REGION,TRAIN_IMAGE_URI,DATA_URI,MODEL_OUTPUT_DIR, and the Vertex endpoint identifiers are used across build, train, deploy, and runtime verification steps.
platform/deployment/local/.env.example, platform/deployment/prod/.env.example, platform/deployment/local/.env, platform/deployment/prod/.env
These files drive the ML lifecycle from the Compose layer. The verification guide uses them to keep build, push, train, deploy, and runtime validation aligned with the same project, region, and registry settings.
| Variable | Role in the Vertex AI lifecycle |
|---|---|
PROJECT_ID |
Used to derive Artifact Registry paths and Vertex resource names |
REGION |
Used for Vertex AI and Artifact Registry host normalization |
DATA_URI |
Training input CSV for the trainer entrypoint |
MODEL_OUTPUT_DIR |
GCS path where model.joblib is written |
TRAIN_IMAGE_URI |
Training container image tag used by Custom Jobs |
PREDICTION_CONTAINER_IMAGE |
Serving container image used during model import/deploy |
VERTEX_CREDIT_ENDPOINT_NAME |
Preferred endpoint resource name for runtime scoring |
VERTEX_CREDIT_ENDPOINT_ID |
Alternate endpoint selector for runtime scoring |
CREDIT_SCORE_PROVIDER |
Must be set to vertex for Vertex-based runtime scoring |
CREDIT_SCORE_ALLOW_HEURISTIC_FALLBACK |
Fail-closed toggle for runtime scoring |
SERVICE_ACCOUNT_EMAIL |
Used by auth helpers when impersonation is needed |
GCP_CREDENTIALS_PATH |
Path to the service account key used by registry helpers |
2. Training Image Preparation
Training Image Build and Push Helper
scripts/vertex_ai/build_and_push_training_image.sh
This helper prepares the Docker image that Vertex AI Custom Jobs execute. The verification guide shows that it wraps registry/auth setup, normalizes TRAIN_IMAGE_URI to match REGION, and then runs Compose build/push so the image tag used by the training job always matches the image that was actually pushed.
| Step | Behavior |
|---|---|
| Registry setup | Calls scripts/vertex_ai/ensure_registry_and_auth.sh first |
| Auth setup | Uses service-key.json by default when present, or impersonates the configured service account |
| Image normalization | Forces the registry host to match the configured REGION |
| Architecture | Forces linux/amd64 during build so Vertex workers can pull the image |
| Output | Pushes us-central1-docker.pkg.dev/${PROJECT_ID}/transunion/pipeline:latest unless overridden |
Registry and Auth Helper
scripts/vertex_ai/ensure_registry_and_auth.sh
This helper prepares Artifact Registry access before the build/push step. The verification guide states that it configures Docker for the registry host derived from TRAIN_IMAGE_URI, creates the repository if missing, and grants writer access to the selected service account.
3. Vertex AI Pipeline Orchestrator
Vertex AI Pipeline Runner
backend/src/credit-score-pipeline/run_vertex_pipeline.py
This module is the operational center of the Vertex AI workflow. The tests in backend/tests/usecase/test_vertex_pipeline_machine_retry.py verify the behaviors that matter for deployment resilience: machine-type retry, error augmentation, predict-only execution, and endpoint-based runtime checks.
| Method | Description |
|---|---|
run_training_job |
Submits the Custom Job and retries across TRAIN_MACHINE_TYPES until a memory-safe machine succeeds |
upload_model |
Uploads the trained artifact to Vertex AI for registration |
deploy_model |
Deploys the uploaded model to an endpoint and retries across DEPLOY_MACHINE_TYPES |
predict_example |
Sends a smoke prediction to the deployed endpoint and retries when the serving contract rejects the first payload |
resolve_existing_model |
Resolves an already-registered model when the run is predict-only or skips retraining |
resolve_existing_endpoint |
Resolves an already-deployed endpoint for runtime verification |
main |
Orchestrates the enabled pipeline steps controlled by the run flags |
_is_memory_error |
Detects memory-related job failures |
_augment_training_error |
Adds actionable context to training failures |
_is_instance_type_error |
Detects dict-to-float serving-contract errors |
_is_feature_count_mismatch_error |
Detects sklearn feature-count mismatch errors |
_expected_feature_count |
Extracts the expected input width from sklearn error text |
_resize_numeric_instance |
Pads or trims numeric vectors to the expected width |
_to_numeric_instance |
Filters mixed payloads down to numeric values |
Trainer Entrypoint
backend/src/credit-score-pipeline/junya_credit_score_model.py
This is the entrypoint invoked by the Vertex training container. The verification guide shows the runtime command pattern:
python ../../backend/src/vertex_ai_pipeline/junya_credit_score_model.py \
--data_uri "$DATA_URI" --model_output_dir "$MODEL_OUTPUT_DIR"
The documented container flow uses the Dockerfile ENTRYPOINT ["python", "junya_credit_score_model.py"], so the console path leaves the command empty and passes only arguments.
4. Runtime Endpoint Configuration
Vertex Runtime Provider
backend/src/usecase/credit_score_runtime.py
This module is the runtime bridge that sends the application’s feature payload to a Vertex endpoint. It resolves project and endpoint identity from env vars, flattens the feature payload into a sklearn-safe numeric shape, calls the Vertex endpoint, and validates the returned contract.
| Method | Description |
|---|---|
_require_project_id |
Reads PROJECT_ID or GCP_PROJECT_ID and raises if neither is present |
_resolve_endpoint_name |
Builds a full Vertex endpoint resource name from VERTEX_CREDIT_ENDPOINT_NAME or VERTEX_CREDIT_ENDPOINT_ID |
_flatten_features_for_vertex |
Converts nested and mixed-type feature data into a flat numeric dictionary |
_score_with_vertex |
Initializes Vertex AI and calls endpoint.predict |
score_housing_application |
Returns the application’s credit-scoring result using the configured provider |
Runtime Endpoint Resolution
The endpoint resolver accepts either a full resource name or a short ID.
| Input | Result |
|---|---|
projects/.../locations/.../endpoints/... |
Used as-is |
123456 |
Expanded to projects/{project_id}/locations/{region}/endpoints/123456 |
Runtime Contract Inputs
The verification guide requires these runtime variables:
CREDIT_SCORE_PROVIDER=vertexPROJECT_IDorGCP_PROJECT_IDREGIONVERTEX_CREDIT_ENDPOINT_NAMEorVERTEX_CREDIT_ENDPOINT_ID
Optional fail-closed behavior is controlled by:
CREDIT_SCORE_ALLOW_HEURISTIC_FALLBACK=false
5. Verification and Smoke Test Helpers
Fast Pipeline Smoke Check
scripts/vertex_ai/fast_credit_pipeline_check.sh
This helper performs a quick PASS/FAIL validation after an endpoint already exists. The verification guide states that it:
- Pulls
TRAIN_IMAGE_URI - Runs a small training-image import check using
ExtraTreesClassifier - Runs runtime endpoint verification through
verify_runtime_credit_scoring.sh
Useful toggles documented in the runbook:
| Variable | Effect |
|---|---|
IMAGE_IMPORT_CHECK=0 |
Skips the training-image import sanity check |
VERIFY_STRICT=1 |
Fails when endpoint traffic or predict readiness is missing |
WAIT_FOR_TRAFFIC_SPLIT_SECONDS=300 |
Adjusts how long the helper waits for traffic split readiness |
VERIFY_ALLOW_SHAPE_MISMATCH=0 |
Fails on known serving-contract mismatches |
FAST_CHECK_REPORT_PATH |
Writes a final report file |
FAIL_ON_PREDICT_ERROR |
Controls whether predict smoke failures are fatal |
The guide also states that the helper ends with a machine-readable status line:
FAST_CHECK_RESULT=PASSFAST_CHECK_RESULT=WARNFAST_CHECK_RESULT=FAIL
Runtime Credit Scoring Verifier
scripts/vertex_ai/verify_runtime_credit_scoring.sh
This helper validates the live Vertex endpoint separately from retraining. The guide states that it prefers the Python SDK (google-cloud-aiplatform) and falls back to gcloud ai endpoints predict when the SDK is unavailable.
| Behavior | Details |
|---|---|
| Endpoint discovery | Uses VERTEX_CREDIT_ENDPOINT_NAME or VERTEX_CREDIT_ENDPOINT_ID |
| Traffic handling | Auto-selects a deployed endpoint with active traffic when needed |
| Strict mode | Waits longer and fails when traffic never becomes ready |
| Predict shape handling | Can fail hard or continue with warning for known contract mismatches |
Training and Deployment Workflow
Build the Training Image
The training image is built and pushed before the pipeline run so Vertex Jobs consume the exact image version that was validated locally. The runbook explicitly says this prevents stale image usage and region mismatch issues.
cd "$REPO_ROOT"
scripts/vertex_ai/build_and_push_training_image.sh platform/deployment/local/.env
Key behaviors from the runbook:
- It is safe to pass a different env file for prod.
- If
TRAIN_IMAGE_URIis omitted, the compose stack derives a tag usingPROJECT_ID. - If the registry host in
TRAIN_IMAGE_URIdoes not matchREGION, the helper normalizes it. - The build is forced to
linux/amd64to match Vertex worker architecture.
Run the Vertex AI Pipeline
The orchestrator runs through Docker Compose so the training container and the orchestration environment stay in sync.
BUILD_BEFORE_RUN=1 scripts/vertex_ai/run_pipeline_via_compose.sh platform/deployment/local/.env
The guide documents three step gates:
RUN_TRAIN_STEPRUN_UPLOAD_STEPRUN_DEPLOY_STEPRUN_PREDICT_STEP
It also documents the failure toggle:
FAIL_ON_PREDICT_ERROR
Operational output from the runner includes:
- Custom Job resource name
- Model upload resource name
- Endpoint resource name
- Prediction response
Register and Deploy in Vertex AI
The console-only path mirrors the automated workflow:
- Launch a Vertex AI Custom Training job.
- Confirm
model.joblibexists inMODEL_OUTPUT_DIR. - Import the model from GCS.
- Deploy an endpoint.
- Run a test prediction.
The runbook explicitly uses:
- training image URI from
TRAIN_IMAGE_URI - artifact URI like
gs://transuniondataset/related_files_we_made/models/transunion_v1 - serving container
us-docker.pkg.dev/vertex-ai/prediction/sklearn-cpu.1-4:latest
Verify the Result
The verification doc describes the expected end state:
model.joblibexists in GCS under the configured model output directory- Vertex AI Custom Jobs shows the job as successful
- Vertex AI Endpoints shows healthy replicas
- The smoke prediction returns a
predictionsarray
Retry and Error Handling
Training Job Retry Semantics
The tests in backend/tests/usecase/test_vertex_pipeline_machine_retry.py show that run_training_job retries machine types in order until the job succeeds.
| Rule | Behavior |
|---|---|
| Memory-related failure | Retry on the next TRAIN_MACHINE_TYPES entry |
| Non-memory failure | Re-raise the error |
| Binary I/O import failure | _augment_training_error adds a filesystem-read hint and points to build_and_push_training_image.sh |
The helper recognizes these memory-related patterns:
code: 8Resource exhausted while scheduling
It also surfaces a known sklearn binary error as a filesystem-read problem so the operator can rebuild the training image.
Deployment Retry Semantics
deploy_model retries across DEPLOY_MACHINE_TYPES when deployment fails for known capacity-related reasons.
| Failure pattern | Observed behavior |
|---|---|
| Model server out of memory | Retries with the next deployment machine type |
| Quota exhausted | Retries with the next deployment machine type |
| Other runtime error | Re-raises with machine-type context |
The tests verify that the helper keeps advancing until a machine type succeeds.
Prediction Smoke Test Handling
The smoke prediction step is intentionally tolerant of contract mismatches when configured to be non-fatal.
| Case | Behavior |
|---|---|
FAIL_ON_PREDICT_ERROR=0 |
Logs a skipped smoke test and continues |
FAIL_ON_PREDICT_ERROR=1 |
Re-raises the prediction failure |
float() argument must be a string or a real number, not 'dict' |
Detected as a Vertex serving-contract mismatch |
| Feature-count mismatch | The helper can resize the numeric vector and retry |
The tests show three retry shapes in the prediction path:
- First attempt with the raw dict payload
- Second attempt with a numeric vector
- Third attempt with a resized vector when a feature-count mismatch is detected
Known Error Patterns Covered by Tests
| Helper | Pattern it matches |
|---|---|
_is_instance_type_error |
Wrapped dict to numeric conversion failures |
_is_feature_count_mismatch_error |
sklearn feature-count mismatch strings and wrapped payloads |
_expected_feature_count |
Extracts the expected input width from sklearn error text |
_resize_numeric_instance |
Pads a numeric list to the expected feature width |
Verification Runbooks
1. Quick End-to-End Smoke Check
[!NOTE] The tests show that a deploy-time or predict-time mismatch is treated as a serving-contract issue, not as a transport issue. The runtime helper distinguishes these cases so operational failures are reported with the right retry or skip behavior.
Run this when you only need a fast verification against an existing endpoint.
scripts/vertex_ai/fast_credit_pipeline_check.sh platform/deployment/local/.env
What this checks:
- training image import sanity
- endpoint readiness
- runtime prediction path
- final report output
2. Runtime-Only Verification
Run this when the model is already deployed and you only want to validate endpoint readiness.
VERIFY_STRICT=1 WAIT_FOR_TRAFFIC_SPLIT_SECONDS=600 \
scripts/vertex_ai/verify_runtime_credit_scoring.sh platform/deployment/local/.env
The guide says this:
- waits for traffic split readiness
- performs
endpoint.predict - fails in strict mode if readiness never arrives
3. Full Pipeline Run
Run this when you need the entire train → upload → deploy → predict sequence.
BUILD_BEFORE_RUN=1 scripts/vertex_ai/run_pipeline_via_compose.sh platform/deployment/local/.env
The runbook also documents:
BUILD_BEFORE_RUN=0to skip rebuildingPUSH_IMAGE_BEFORE_RUN=0to skip pushingRUN_DEPLOY_STEP=0to stop after model uploadRUN_PREDICT_STEP=0to skip the smoke prediction
4. Console-Only Validation
Use the Vertex UI if you want to verify the path manually.
{
"income_value": 120000.0,
"savings_value": 50000.0,
"budget_value": 400000.0,
"coverage_score": 0.58,
"days_covered": 92,
"salary_detected": 1.0,
"pd_hint": "endpoint should respond with pd_p50_12m/pd_p90_12m contract"
}
The runbook says that a predictions array confirms the end-to-end path is working.
5. Cleanup
After validation, delete the endpoint if it is no longer needed.
gcloud ai endpoints list --region "$REGION"
gcloud ai endpoints delete ENDPOINT_ID --region "$REGION"
Sequence Flows
Build, Train, Upload, Deploy, and Predict
sequenceDiagram
participant Op as Operator
participant Script as run_pipeline_via_compose.sh
participant Compose as Docker Compose
participant Runner as run_vertex_pipeline.py
participant Job as Vertex Custom Job
participant Reg as Model Registry
participant Ep as Vertex Endpoint
Op->>Script: start pipeline with deployment env file
Script->>Compose: build and push training image
Compose->>Runner: start orchestration with env settings
Runner->>Job: submit training job
Job-->>Runner: training artifact written to MODEL_OUTPUT_DIR
Runner->>Reg: upload model artifact
Reg-->>Runner: model resource name
Runner->>Ep: deploy model to endpoint
Ep-->>Runner: endpoint resource name
Runner->>Ep: run smoke prediction
Ep-->>Runner: predictions array or contract error
Runner-->>Op: logs resource names and final status
Runtime Endpoint Verification
sequenceDiagram
participant Op as Operator
participant Verify as verify_runtime_credit_scoring.sh
participant SDK as google-cloud-aiplatform or gcloud
participant Ep as Vertex Endpoint
participant App as credit_score_runtime.py
Op->>Verify: run runtime verification with env file
Verify->>SDK: initialize project and location
Verify->>Ep: predict with flattened feature payload
Ep-->>Verify: prediction response
Verify-->>Op: readiness result and contract status
App->>Ep: score_housing_application uses same endpoint settings
Training Retry and Fallback Flow
flowchart TD
Start([run_training_job])
TryMachine[Select machine type from TRAIN_MACHINE_TYPES]
RunJob[Submit Custom Job]
MemoryError{Memory related error}
Success([Job succeeds])
Augment[Augment error message]
NextMachine[Try next machine type]
Fail([Raise error])
Start --> TryMachine
TryMachine --> RunJob
RunJob --> MemoryError
MemoryError -->|yes| NextMachine
NextMachine --> TryMachine
MemoryError -->|no| Augment
Augment --> Fail
RunJob --> Success
Environment and Lifecycle Participation
Local and Prod Env Files
The verification guide explicitly uses deployment-level env files so the same pipeline can run against the local or prod Docker Compose stack.
| File | Role |
|---|---|
platform/deployment/local/.env.example |
Starter file for local Vertex AI configuration |
platform/deployment/prod/.env.example |
Starter file for production Vertex AI configuration |
platform/deployment/local/.env |
Actual local run configuration |
platform/deployment/prod/.env |
Actual production run configuration |
Lifecycle Flow
- Populate the env file.
- Authenticate with Google Cloud.
- Configure Docker for Artifact Registry.
- Build and push the training image.
- Run the pipeline via Compose.
- Verify the endpoint.
- Clean up the endpoint when done.
Credential and Registry Participation
The verification guide supports both:
- a repo-root service key at
./service-key.json - impersonation of
[email protected]
This lets the same helpers run in local and CI-like contexts without changing the pipeline contract.
Testing Considerations
The tests in backend/tests/usecase/test_vertex_pipeline_machine_retry.py encode the expected operational behavior for the pipeline.
| Scenario | What is verified |
|---|---|
| Memory-heavy training failure | Retries to the next training machine type |
| Non-memory training error | Propagates the error |
| Invalid endpoint env | resolve_existing_endpoint requires EXISTING_ENDPOINT_NAME |
| Deployment quota exhaustion | Retries to the next deployment machine type |
| Deployment OOM | Retries to the next deployment machine type |
| Predict-only mode | Uses the existing endpoint without retraining or redeploying |
| Fatal predict mode | Re-raises prediction errors when FAIL_ON_PREDICT_ERROR=True |
| Non-fatal predict mode | Logs a skipped smoke test and continues |
| Shape mismatch | Resizes the numeric vector and retries |
| Wrapped dict-to-float error | Recognized as an instance-type contract issue |
Key Classes Reference
| Class | Responsibility |
|---|---|
run_vertex_pipeline.py |
Orchestrates training, model upload, endpoint deployment, and smoke prediction with retry handling |
junya_credit_score_model.py |
Training entrypoint executed inside the Vertex AI Custom Job container |
credit_score_runtime.py |
Resolves the Vertex endpoint and scores applications at runtime |
vertex_ai_verification.md |
Runbook for registry auth, image preparation, pipeline execution, and verification |
test_vertex_pipeline_machine_retry.py |
Verifies retry, smoke-test, and contract-mismatch behavior |
build_and_push_training_image.sh |
Builds and pushes the training image with normalized registry settings |
run_pipeline_via_compose.sh |
Runs the full pipeline through Docker Compose |
verify_runtime_credit_scoring.sh |
Validates the live Vertex endpoint contract |
fast_credit_pipeline_check.sh |
Runs the fast image-and-endpoint smoke check |