Credit Scoring and Pipeline Domain

Overview

This domain covers the full credit-score lifecycle used by Dwilar’s housing flow: model training, deterministic completeness gating, Vertex AI deployment, and deployed-endpoint smoke verification. It separates “should we score this application?” from “can the training and serving artifact be deployed correctly?”, so incomplete or high-risk cases are blocked before model execution and the deployment pipeline can be validated independently.

The code in this section centers on the unified credit model in backend/src/ml/credit_model.py, the gating rules in backend/src/ml/credit_pipeline_validation.py, and the Vertex AI orchestration and verification workflow documented in documents/vertex_ai_verification.md and exercised by backend/tests/usecase/test_vertex_pipeline_machine_retry.py. Runtime contract checks in backend/tests/usecase/test_credit_score_runtime.py and backend/tests/usecase/test_credit_scoring.py ensure that trained artifacts and deployed endpoints return the expected scoring shape and fail closed on contract mismatches.

Architecture Overview

flowchart TB
    subgraph TrainingAndValidation [Training and Validation]
        CanonicalInput[Canonical credit input]
        PipelineValidation[credit_pipeline_validation.py]
        CreditModel[credit_model.py]
        VertexTraining[credit_score_training.py]
        CanonicalInput --> PipelineValidation
        PipelineValidation -->|pass| CreditModel
        PipelineValidation -->|refer or decline| ManualReview[Manual review or stop]
        CreditModel --> VertexTraining
    end

    subgraph VertexOrchestration [Vertex AI Orchestration]
        RunPipeline[run_vertex_pipeline.py]
        BuildPush[Build and push training image]
        CustomJob[Vertex Custom Job]
        ModelArtifact[model.joblib]
        DeployEndpoint[Vertex endpoint deploy]
        SmokePredict[Smoke prediction]
        RunPipeline --> BuildPush
        BuildPush --> CustomJob
        CustomJob --> ModelArtifact
        ModelArtifact --> DeployEndpoint
        DeployEndpoint --> SmokePredict
    end

    subgraph RuntimeContract [Runtime Contract and Verification]
        RuntimeScore[credit_score_runtime.py]
        VerifyGuide[vertex_ai_verification.md]
        DeployedEndpoint[Deployed Vertex endpoint]
        RuntimeScore --> DeployedEndpoint
        VerifyGuide --> DeployedEndpoint
        SmokePredict --> VerifyGuide
    end

    CreditModel --> ModelArtifact
    PipelineValidation --> RuntimeScore

Component Structure

1. Training Model Library

credit_model.py

backend/src/ml/credit_model.py

This module defines the unified credit-score training flow used by the pipeline. It loads domestic and foreign datasets, builds the feature matrix, trains the model ensemble, computes validation metrics, and writes the local artifact to UNIFIED_MODEL_PATH.

Key module constants

Constant Type Description
RANDOM_SEED int Fixed seed used by the training flow.
TARGET_COL str Primary score target name: VantageScore.
FOREIGN_TARGET_COL str Alternate target name: dwilar_risk_score.
DOMESTIC_URI str Default GCS URI for the domestic training dataset.
FOREIGN_URI str Default GCS URI for the foreign training dataset.
OPEN_BANKING_PREFIX str GCS prefix for open-banking inputs.
CUSTOMER_FLAGS_PREFIX str GCS prefix for customer flags.
ARTIFACT_DIR Path Local artifact directory for training assets.
MODELS_DIR Path Local directory where model artifacts are written.
UNIFIED_MODEL_PATH Path Local serialized model path, data/models/credit_score_model.pkl.
OPEN_BANKING_LEAKAGE_TOKENS tuple[str, ...] Leakage markers: target, label, outcome.
OPEN_BANKING_LGBM_PARAMS dict[str, object] LightGBM parameter set for the open-banking model path.
BASE_FEATURE_COLUMNS list[str] Renamed canonical bureau columns excluding the target.
ENGINEERED_COLUMNS list[str] Engineered bureau features such as utilization and trade-age aggregates.
FEATURE_COLUMNS list[str] Bureau feature set used by the model.
ALL_FEATURE_COLUMNS list[str] Bureau features plus open-banking features.

Public methods

Method Description
build_features Transforms a dataframe into model-ready features and target output.
train_unified_credit_model Trains the unified model and returns the trained models plus metrics.
report_accuracy Prints the training and validation quality summary.
main Command-line entrypoint for local or containerized training.

Training behavior

  • build_features constructs derived credit features such as balance_total, num_tradelines, util_total, loan_ratio, debt_burden_score, and credit-history span metrics.
  • The training flow saves a serialized artifact with joblib.dump(...) to UNIFIED_MODEL_PATH.
  • main accepts --data-uri, --extra-uri, and --metrics-json; when --metrics-json is set, the metrics dictionary is written to disk as JSON.

2. Pipeline Gating and Completeness Rules

credit_pipeline_validation.py

[!NOTE] OPEN_BANKING_LEAKAGE_TOKENS is part of the feature-hygiene contract in this module and is used to keep target-like columns out of the model input.

backend/src/ml/credit_pipeline_validation.py

This module decides whether an application can be scored and returns deterministic gating results. It computes completeness from canonical input, classifies blocker flags, and applies the decline → refer → pass priority order.

Key module constants

Constant Type Description
MIN_DAYS_COVERED int Minimum open-banking history required to avoid the insufficient-history flag.
BUREAU_MAX_AGE_DAYS float Maximum acceptable bureau report age before recency is considered stale.
MIN_COMPLETENESS_SCORE_TO_PASS float Minimum completeness score required to avoid a refer outcome.
_BUREAU_REQUIRED_PRODUCTS frozenset[str] Product types that require bureau data, defaulting to high_ticket_loan.
_DECLINE_FLAGS frozenset[str] Hard blockers: KYC_NOT_PASSED, FRAUD_RISK_DETECTED, AML_RISK_DETECTED.
_REFER_FLAGS frozenset[str] Soft blockers: BUREAU_REQUIRED_BUT_ABSENT, NO_BANKING_DATA, INSUFFICIENT_HISTORY.

Public methods

Method Description
compute_completeness Builds a completeness report from the canonical credit input.
gate Attaches the report to canonical.data_completeness and returns the gating result.

Observed gating result fields

Field Type Description
gate_result Literal["pass", "refer", "decline"] Final outcome derived from flags and completeness score.
blocking_flags list[str] Flags that block model scoring immediately.
warn_flags list[str] Flags that do not block scoring.
completeness_score float Weighted completeness score used in the final decision.

Observed completeness report fields used by gate()

Field Type Description
kyc_passed bool KYC status used in the weighted score.
banking_available bool Indicates whether banking data is present.
days_covered_ok bool Indicates whether the minimum history threshold was met.
salary_detected bool Signals income verification from banking activity.
income_verified bool Composite verification flag from salary, bureau, or stated income.
bureau_available bool Indicates whether bureau data is present.
bureau_recency_ok bool Indicates whether bureau data is sufficiently fresh.
flags list[str] Derived blocker and warning flags.
gate_result str Final gate output.
completeness_score float Weighted completeness score.

Gating logic

  • KYC_NOT_PASSED, FRAUD_RISK_DETECTED, and AML_RISK_DETECTED force decline.
  • BUREAU_REQUIRED_BUT_ABSENT, NO_BANKING_DATA, and INSUFFICIENT_HISTORY force refer.
  • A completeness score below MIN_COMPLETENESS_SCORE_TO_PASS also forces refer.
  • Otherwise the result is pass.
stateDiagram-v2
    [*] --> pass
    pass --> decline: KYC_NOT_PASSED
    pass --> decline: FRAUD_RISK_DETECTED
    pass --> decline: AML_RISK_DETECTED
    pass --> refer: BUREAU_REQUIRED_BUT_ABSENT
    pass --> refer: NO_BANKING_DATA
    pass --> refer: INSUFFICIENT_HISTORY
    pass --> refer: completeness below threshold
    decline --> [*]
    refer --> [*]
    pass --> [*]

3. Vertex AI Training Entry Point

credit_score_training.py

[!NOTE] gate() mutates the input object by assigning the computed report to canonical.data_completeness, so downstream feature code can reuse the same report without recomputing it.

backend/src/credit-score-pipeline/credit_score_training.py

This script is the Vertex-facing training entrypoint used by the deployment workflow. The documented contract is that it consumes the training data URI and model output directory, trains the credit model, and writes a single serving artifact as model.joblib into MODEL_OUTPUT_DIR on GCS.

Observed training contract from the verification guide

Item Value
Training input argument --data_uri
Model output argument --model_output_dir
Artifact name model.joblib
Output location MODEL_OUTPUT_DIR on GCS
Serving compatibility Vertex prebuilt sklearn container contract

Validation and robustness behavior documented for the training path

  • Score-like columns are auto-detected, including names such as target, dwilar_risk_score, vantage_score, and credit_score.
  • Training uses numeric-only features with NaN and infinity cleanup, and the feature matrix is downcast to float32.
  • MAX_TRAIN_ROWS caps the sample size for large CSVs.
  • FEATURE_COLUMNS_LIMIT constrains the feature set to the highest-variance numeric columns.
  • Single-class targets are handled with constant fallback classifiers.
  • The pipeline can skip unstable validation splits for tiny or skewed datasets.
  • Unexpected model or preprocessing failures trigger an emergency constant-model artifact fallback.

Documented configuration knobs

Variable Description
TARGET_COLUMN Forces the target column when it is not auto-detected.
TARGET_SCORE_THRESHOLD Threshold used to binarize continuous targets.
MAX_TRAIN_ROWS Row sampling cap for large datasets.
FEATURE_COLUMNS_LIMIT Limit on selected numeric features.
DEPLOY_MACHINE_TYPE Default deployment shape for the Vertex endpoint.

4. Vertex AI Orchestration Workflow

run_vertex_pipeline.py

backend/src/credit-score-pipeline/run_vertex_pipeline.py

This script orchestrates the end-to-end Vertex AI workflow: build and push the training image, run the custom training job, upload the resulting model, deploy an endpoint, and optionally issue a smoke prediction. The tests show that it also contains retry and error-classification logic for machine sizing and prediction mismatch failures.

Public and orchestration methods

Method Description
run_training_job Submits the Vertex CustomJob and retries across configured machine types.
upload_model Registers the trained artifact into Vertex Model Registry.
deploy_model Deploys the registered model to an endpoint.
predict_example Sends a smoke prediction request to the deployed endpoint.
resolve_existing_model Resolves a model for predict-only or post-deploy flows.
resolve_existing_endpoint Resolves an endpoint for predict-only or post-deploy flows.
main Orchestrates the selected pipeline steps based on runtime flags.
_is_memory_error Detects memory-related failures and schedules machine retry.
_augment_training_error Adds actionable context to training failures.
_is_feature_count_mismatch_error Detects serving-contract feature-count mismatches.
_expected_feature_count Extracts the expected feature size from an error.
_resize_numeric_instance Pads or truncates numeric smoke-test input to the expected size.

Behavior confirmed by tests

  • run_training_job retries machine types in TRAIN_MACHINE_TYPES order until a job succeeds.
  • _is_memory_error matches code: 8 and Resource exhausted style messages.
  • _augment_training_error adds a filesystem-read-error explanation and points back to build_and_push_training_image.sh.
  • _is_feature_count_mismatch_error recognizes wrapped Vertex and sklearn feature-mismatch payloads.
  • _expected_feature_count extracts the expected feature count from the exception text.
  • _resize_numeric_instance pads the smoke-test vector to the requested length.
  • main can run in predict-only mode without retraining, uploading, or deploying.

Runtime flags documented in the orchestration guide

Flag Purpose
BUILD_BEFORE_RUN Rebuilds the training image before orchestration.
PUSH_IMAGE_BEFORE_RUN Pushes the image before the Vertex job starts.
RUN_DEPLOY_STEP Skips deployment when set to 0.
RUN_PREDICT_STEP Skips the smoke prediction when set to 0.
FAIL_ON_PREDICT_ERROR Makes smoke-predict failures fatal when set to 1.
sequenceDiagram
    participant O as Operator
    participant M as run_vertex_pipeline.main
    participant T as run_training_job
    participant A as upload_model
    participant D as deploy_model
    participant P as predict_example
    participant V as Vertex AI

    O->>M: Start pipeline
    M->>T: Submit CustomJob
    T->>V: Run training image
    V-->>T: Training result and artifact
    T-->>M: Job reference
    M->>A: Register model artifact
    A->>V: Import model from GCS
    V-->>A: Model resource
    A-->>M: Model reference
    M->>D: Deploy endpoint
    D->>V: Create endpoint and replicas
    V-->>D: Endpoint resource
    D-->>M: Endpoint reference
    M->>P: Smoke predict
    P->>V: Predict request
    V-->>P: Predictions response
    P-->>M: Smoke result

Machine retry and smoke-predict handling

  • Memory-related training failures are retried across the configured machine list.
  • Feature-count mismatches during smoke prediction can be treated as a skippable verification outcome when FAIL_ON_PREDICT_ERROR=0.
  • When FAIL_ON_PREDICT_ERROR=1, the same prediction failure is re-raised and the orchestration exits non-zero.

5. Vertex AI Verification Guide

vertex_ai_verification.md

documents/vertex_ai_verification.md

This document is the operational runbook for training, registering, deploying, and verifying the model in Vertex AI. It covers both the full pipeline and the faster smoke-check path used after deployment.

Documented workflow steps

  1. Prepare deployment .env values, including PROJECT_ID, REGION, DATA_URI, MODEL_OUTPUT_DIR, and TRAIN_IMAGE_URI.
  2. Authenticate with Google Cloud and configure Docker for Artifact Registry.
  3. Run the optional local training smoke test.
  4. Build and push the training image.
  5. Run the full Vertex AI pipeline.
  6. Confirm model.joblib exists and check the Custom Job and Endpoint state in the console.
  7. Use the console-only path when needed.
  8. Verify the deployed runtime endpoint independently.

Documented smoke-verification commands

scripts/vertex_ai/fast_credit_pipeline_check.sh platform/deployment/local/.env
scripts/vertex_ai/verify_runtime_credit_scoring.sh platform/deployment/local/.env

Verification toggles

Variable Purpose
IMAGE_IMPORT_CHECK Skips the Docker image import check when set to 0.
VERIFY_STRICT Enables strict endpoint readiness and failure behavior.
WAIT_FOR_TRAFFIC_SPLIT_SECONDS Controls endpoint-readiness wait time.
VERIFY_ALLOW_SHAPE_MISMATCH Allows known serving-contract mismatches to continue with a warning.
AUTO_SELECT_DEPLOYED_ENDPOINT Auto-selects a recently updated endpoint with traffic split.
VERIFY_PREDICT Skips the runtime prediction call when set to 0.

Artifact and deployment expectations

Item Expected value
Model artifact directory gs://transuniondataset/related_files_we_made/models/transunion_v1
Model artifact file model.joblib
Serving container us-docker.pkg.dev/vertex-ai/prediction/sklearn-cpu.1-4:latest
Console deploy machine n1-standard-2
Replica range Min 1, max 3

Smoke-predict expectation

  • A successful test prediction returns a predictions array.
  • A response that behaves like 'dict' object has no attribute 'predict' is treated as a serving-contract mismatch rather than a network failure.
  • When strict mode is off, the verifier can continue with a warning instead of failing the pipeline.

6. Runtime Score Contract Used by Verification

credit_score_runtime.py

[!IMPORTANT] The guide distinguishes between the training artifact directory used for model registration and the runtime endpoint used for prediction verification. The deployment succeeds only when the uploaded artifact layout matches the prebuilt sklearn serving contract.

backend/src/usecase/credit_score_runtime.py

This runtime module is the contract consumer used by the smoke-verification path. It builds the feature payload, calls Vertex AI, validates the returned payload, and converts it into the application-facing score estimate used by the housing flow.

Observed data contract class

Property Type Description
pd_p50_12m float Median predicted default probability.
pd_p90_12m float Upper-tail predicted default probability.
reason_codes list[str] Up to six approved explanation codes.
breakdown Dict[str, float] Required score breakdown with fixed keys.
model_version str Model identifier returned in meta.model_version.

Observed application estimate class

Property Type Description
score int Final bounded score.
score_band str Score band label.
provider str Provider name used by the scorer.
model_version str Model version returned in the estimate.
featureset_version str Feature-set identifier used by the scorer.
calibration_version str Calibration version applied to the score.
confidence float Confidence score for the result.
data_coverage Dict[str, Any] Coverage metrics returned with the score.
reason_codes list[str] Explanation codes carried through from the engine response.
breakdown Dict[str, float] Score breakdown returned by the engine.
lita_global_score int Internal global score representation.
global_band str Internal banding for the global score.
risk_flags list[dict[str, str]] Structured risk flags.
target_scale `str \ None`
target_score `int \ None`
target_confidence `float \ None`

Contract validation behavior

  • The Vertex response must be an object.
  • pd_p50_12m must be numeric and within [0, 1].
  • pd_p90_12m must be numeric, at least pd_p50_12m, and at most 1.
  • reason_codes must be a list of strings and is capped at six items.
  • breakdown must contain exactly the expected keys and numeric values.
  • meta.model_version is required.
  • Invalid payloads raise ValueError and fail closed.

Runtime flow

sequenceDiagram
    participant S as score_housing_application
    participant B as _score_with_vertex
    participant E as Vertex Endpoint
    participant V as _validate_risk_engine_contract

    S->>B: Flatten features
    B->>E: endpoint.predict(instances)
    E-->>B: predictions
    B->>V: Validate first prediction
    V-->>B: _RiskEngineOutput
    B-->>S: Risk engine output

Training Artifact Format

The deployment workflow uses two artifact conventions:

Artifact Location Purpose
credit_score_model.pkl backend/src/ml/credit_model.py local path via UNIFIED_MODEL_PATH Local serialized training output from the model library.
model.joblib GCS MODEL_OUTPUT_DIR Vertex-serving artifact expected by the prebuilt sklearn container.

The verification guide treats model.joblib as the deployable unit. Console import uses the artifact directory, not the file name, and the deployed endpoint is validated with a small smoke prediction after deployment.

Completeness Gating Rules

[!NOTE] The local library artifact path and the Vertex-serving artifact name are intentionally different in the documented workflow: the training library writes a local pickle, while the Vertex entrypoint produces model.joblib for deployment.

The gating module enforces the following deterministic rule set:

  • KYC_NOT_PASSED, FRAUD_RISK_DETECTED, AML_RISK_DETECTED are hard blockers.
  • BUREAU_REQUIRED_BUT_ABSENT, NO_BANKING_DATA, INSUFFICIENT_HISTORY are refer blockers.
  • The completeness score is a weighted sum of KYC, banking availability, history coverage, salary detection, income verification, bureau availability, and bureau recency.
  • A score below MIN_COMPLETENESS_SCORE_TO_PASS converts the result to refer.
  • Product-specific bureau requirements are checked via _BUREAU_REQUIRED_PRODUCTS.

Gate priority

  1. Decline flags.
  2. Refer flags.
  3. Completeness threshold.
  4. Pass.

This ordering is implemented in _gate_from_flags and exercised by backend/tests/ml/test_pipeline_validation.py.

Open-Banking Feature Alignment

The training and runtime paths share the same feature contract for open-banking inputs. In credit_model.py, ALL_FEATURE_COLUMNS is defined as bureau-derived features plus OPEN_BANKING_FEATURE_COLUMNS, and the code comments state that the list must stay in sync with the Vertex runtime feature builders.

Alignment signals visible in the code

  • Training expects the same open-banking family used by runtime scoring.
  • Runtime builds coverage, affordability, income, liquidity, bureau, and risk features from linked accounts and transactions.
  • The runtime smoke path uses a single feature instance list, which must preserve the training-time feature order and expected width.
  • The mismatch detector in run_vertex_pipeline.py exists specifically to catch deployed artifacts whose feature count does not match the training contract.

Feature families used by the unified scorer

  • Income and affordability
  • Account and institution linkage
  • Transaction activity and salary regularity
  • Coverage and completeness scoring
  • Debt, rent, and recurring obligation signals
  • Balance and liquidity indicators
  • Bureau strength and recency
  • Host-country and corridor context
  • KYC, fraud, and AML flags

Vertex AI Deployment Workflow

Build and Push

The documented helper scripts/vertex_ai/build_and_push_training_image.sh:

  • normalizes TRAIN_IMAGE_URI to the region host used by Vertex AI,
  • configures Artifact Registry access,
  • creates the repository if missing,
  • and builds/pushes the training image with linux/amd64 on Apple Silicon hosts.

Run the pipeline

The full pipeline helper scripts/vertex_ai/run_pipeline_via_compose.sh:

  • reads the deployment .env,
  • submits the custom job,
  • uploads the model,
  • deploys the endpoint,
  • and optionally issues a smoke prediction.

The guide also documents a repo-root invocation to avoid compose path issues and service-name errors.

Console-only path

The manual Vertex console path uses:

  1. Custom Training job.
  2. Artifact verification in GCS.
  3. Model import into Vertex AI.
  4. Endpoint deployment.
  5. Smoke prediction from the console’s test UI.

Cleanup

The guide includes endpoint cleanup commands so you can remove an idle endpoint after verification.

Testing Considerations

backend/tests/ml/test_pipeline_validation.py

This file validates:

  • decline on KYC failure,
  • decline on fraud and AML flags,
  • refer on missing banking data,
  • refer on insufficient history,
  • refer on high_ticket_loan without bureau data,
  • pass for a fully populated application,
  • completeness score weighting,
  • stale bureau behavior,
  • salary detection and income verification combinations,
  • and the side effect that writes the report to canonical.data_completeness.

backend/tests/usecase/test_credit_score_runtime.py

This file validates the runtime contract:

  • the Vertex provider returns the expected score structure,
  • the score remains within the bounded range,
  • the provider is labeled lita,
  • the version fields are preserved,
  • the breakdown and coverage keys are complete,
  • and invalid contract payloads fail closed.

backend/tests/usecase/test_credit_scoring.py

This file validates the heuristic score path:

  • scores stay within the 300–850 bounds,
  • the model version reflects the selected heuristic model,
  • the breakdown contains the expected components,
  • student applications route to the student-specific model,
  • and extreme inputs clamp to the maximum score.

backend/tests/usecase/test_vertex_pipeline_machine_retry.py

This file validates orchestration and smoke behavior:

  • training retries across machine types,
  • memory error detection,
  • training-error augmentation,
  • expected feature-count parsing,
  • vector resizing for smoke checks,
  • mismatch detection from wrapped Vertex responses,
  • predict-only operation with existing resources,
  • and fail-open versus fail-closed prediction smoke handling.

Dependencies

  • Vertex AI SDK through google-cloud-aiplatform
  • Google Cloud CLI and Application Default Credentials
  • Docker and Docker Compose
  • GCS model and dataset URIs
  • Artifact Registry for the training image
  • Training-time Python packages referenced by credit_model.py, including joblib, numpy, pandas, scikit-learn, lightgbm, and xgboost where available

Integration Points

  • Google Cloud Storage for training data and model artifacts.
  • Vertex AI Custom Jobs for training runs.
  • Vertex AI Model Registry for import and versioning.
  • Vertex AI Endpoints for deployment and smoke prediction.
  • Deployment .env files for pipeline configuration.
  • Runtime scoring contract for response-shape validation.

Error Handling

Condition Handling
Missing endpoint name or ID in runtime scoring Raises EnvironmentError in endpoint-name resolution.
Vertex endpoint returns no predictions Raises ValueError before score extraction.
Prediction payload violates the risk-engine contract Raises ValueError and fails closed.
Training worker runs out of memory Retries the job on the next machine type.
Training image or filesystem read error Augments the exception with build/push guidance.
Smoke prediction hits a feature-count mismatch Can be logged and skipped when FAIL_ON_PREDICT_ERROR=0.
Smoke prediction hits a non-mismatch error Can continue or fail based on FAIL_ON_PREDICT_ERROR.

State Management

Pipeline gate states

  • pass
  • refer
  • decline

Vertex job state codes documented in the verification guide

  • 1 = queued
  • 2 = pending or provisioning
  • 3 = running
  • 4 = succeeded
  • 5 = failed

Orchestration control flags

  • RUN_TRAIN_STEP
  • RUN_UPLOAD_STEP
  • RUN_DEPLOY_STEP
  • RUN_PREDICT_STEP
  • FAIL_ON_PREDICT_ERROR

Key Classes Reference

Class Responsibility
credit_model.py Trains the unified credit model, derives features, and writes the local artifact.
credit_pipeline_validation.py Computes completeness and applies deterministic pass, refer, and decline gating.
credit_score_training.py Vertex training entrypoint that writes the deployable model.joblib artifact.
run_vertex_pipeline.py Orchestrates training, model upload, endpoint deployment, and smoke prediction.
vertex_ai_verification.md Operational verification guide for Vertex training, deployment, and runtime smoke checks.
test_pipeline_validation.py Verifies completeness weights, gate priority, and report attachment.
test_credit_score_runtime.py Verifies Vertex runtime contract validation and fail-closed behavior.
test_credit_scoring.py Verifies heuristic scoring bounds and model selection behavior.
test_vertex_pipeline_machine_retry.py Verifies training retries, prediction mismatch handling, and orchestration modes.