Deployment, Operations, and Observability - Environment matrices, secret ownership, and runtime configuration surfaces

Overview

This section maps the repository’s deployment-time and runtime configuration surfaces into one place: local development, production deployment, edge authentication, Vertex AI scoring, and observability. The goal is to show which values are public, which are server-only, which belong in deployment env files, and which are stored as CI/CD secrets or cloud credentials.

The configuration surface spans the Next.js frontend, the FastAPI backend, the Auth0 edge callback and management helper, the Vertex AI pipeline and runtime verifier, and the rollout scripts used in Compose-based deployments. The same application can be run from local .env files, production deployment env files, or GitHub Actions secrets, so the operational boundary is defined primarily by where each variable is consumed.

Architecture Overview

flowchart TB
    subgraph LocalDev[Local Development]
        FrontendEnv[frontend env local example]
        LocalCompose[local docker compose]
        FrontendDev[frontend container]
        BackendDev[backend container]
    end

    subgraph ProdDeploy[Production Deployment]
        ProdEnv[prod env files]
        ProdCompose[prod docker compose]
        FrontendProd[frontend container]
        BackendProd[backend container]
        GCSData[applications json storage]
    end

    subgraph EdgeAuth[Edge Auth]
        Auth0Route[auth callback route]
        Auth0Mgmt[auth0 management helper]
        Auth0Api[Auth0 Management API]
    end

    subgraph VertexAI[Vertex AI Scoring]
        VertexEnv[deployment env files]
        VertexRuntime[credit score runtime helper]
        VertexEndpoint[Vertex AI endpoint]
        VertexVerifier[vertex verification scripts]
    end

    subgraph Observability[Observability]
        SentryEnv[Sentry env vars]
        AmplitudeEnv[Amplitude env vars]
        OTelEnv[OpenTelemetry env vars]
        Logs[Python logging]
    end

    FrontendEnv --> LocalCompose
    LocalCompose --> FrontendDev
    LocalCompose --> BackendDev

    ProdEnv --> ProdCompose
    ProdCompose --> FrontendProd
    ProdCompose --> BackendProd
    ProdCompose --> GCSData

    Auth0Route --> Auth0Api
    Auth0Mgmt --> Auth0Api

    VertexEnv --> VertexRuntime
    VertexVerifier --> VertexRuntime
    VertexRuntime --> VertexEndpoint

    SentryEnv --> FrontendProd
    AmplitudeEnv --> FrontendProd
    OTelEnv --> FrontendProd
    Logs --> BackendProd

Environment Matrix

Frontend build-time and local development variables

[!IMPORTANT] documents/deployment.md shows VITE_BACKEND_API_URL, while frontend/lib/utils.ts reads NEXT_PUBLIC_BACKEND_API_URL. Those are different keys, and the frontend helper only consumes the NEXT_PUBLIC_ name.

frontend/.env.local.example and frontend/Dockerfile

Variable Surface Scope Ownership Consumed by
AUTH0_DOMAIN Frontend auth Server-side auth config Deployment env frontend/lib/auth0-management.ts, Auth0 callback route
AUTH0_CLIENT_ID Frontend auth Server-side auth config Deployment env Auth0 callback route
AUTH0_CLIENT_SECRET Frontend auth Secret Deployment secret storage Auth0 callback route
AUTH0_ORGANIZATION_ID Frontend auth Optional org routing Deployment env Auth0 callback route
NEXT_PUBLIC_AUTH0_ORGANIZATION_ID Frontend auth Client-visible fallback Public runtime env Login redirect logic
AUTH0_SECRET Frontend auth Cookie encryption secret Deployment secret storage Auth0 session handling
AUTH0_BASE_URL Frontend auth Local/app origin Deployment env Auth0 callback route
AUTH0_ISSUER_BASE_URL Frontend auth Tenant issuer origin Deployment env Auth0 callback route
AUTH0_MANAGEMENT_CLIENT_ID Auth0 management Server-only identifier Deployment env frontend/lib/auth0-management.ts
AUTH0_MANAGEMENT_CLIENT_SECRET Auth0 management Secret Deployment secret storage frontend/lib/auth0-management.ts
AUTH0_MANAGEMENT_AUDIENCE Auth0 management API audience Deployment env frontend/lib/auth0-management.ts
NEXT_PUBLIC_BACKEND_API_URL Frontend backend routing Client-visible Build/runtime env frontend/lib/utils.ts, frontend/Dockerfile
NEXT_PUBLIC_AMPLITUDE_API_KEY Analytics Client-visible Public runtime env frontend/Dockerfile, frontend analytics
NEXT_PUBLIC_SENTRY_DSN Observability Client-visible Public runtime env Browser Sentry
SENTRY_DSN Observability Server-side observability Deployment env Server Sentry
NEXT_PUBLIC_SENTRY_ENVIRONMENT Observability Client-visible Public runtime env Browser Sentry
SENTRY_ENVIRONMENT Observability Server-side observability Deployment env Server Sentry
NEXT_PUBLIC_SENTRY_TRACES_SAMPLE_RATE Observability Client-visible Public runtime env Browser Sentry traces
SENTRY_TRACES_SAMPLE_RATE Observability Server-side observability Deployment env Server Sentry traces
NEXT_PUBLIC_SENTRY_REPLAYS_SESSION_SAMPLE_RATE Observability Client-visible Public runtime env Browser replay
NEXT_PUBLIC_SENTRY_REPLAYS_ON_ERROR_SAMPLE_RATE Observability Client-visible Public runtime env Browser replay
SENTRY_AUTH_TOKEN Observability Secret CI/CD secret storage Sentry release upload
SENTRY_ORG Observability Build metadata CI/CD secret storage Sentry release upload
SENTRY_PROJECT Observability Build metadata CI/CD secret storage Sentry release upload
OTEL_SERVICE_NAME Observability Service name Deployment env OpenTelemetry exporter
OTEL_EXPORTER_OTLP_ENDPOINT Observability Collector endpoint Deployment env OpenTelemetry exporter
OTEL_EXPORTER_OTLP_HEADERS Observability Export headers Deployment env or secret store OpenTelemetry exporter

frontend/Dockerfile only forwards NEXT_PUBLIC_BACKEND_API_URL and NEXT_PUBLIC_AMPLITUDE_API_KEY at build time, so changing those values requires rebuilding the image.

Production deployment variables

documents/deployment.md and production Compose expectations

Variable Surface Scope Ownership Consumed by
VITE_BACKEND_API_URL Production frontend docs Public origin setting Deployment env Production frontend container docs
ALLOWED_ORIGINS Production frontend docs CORS/origin allowlist Deployment env Frontend/backend deployment docs
APPLICATIONS_DATA_GCS_BUCKET Application storage Data persistence target Deployment env Frontend application storage route
APPLICATIONS_DATA_GCS_OBJECT Application storage Optional object name Deployment env Frontend application storage route
AUTH0_DOMAIN Production auth Server-side auth config Deployment env injection Auth0 callback and management helper
AUTH0_CLIENT_ID Production auth Public identifier Deployment env injection Auth0 callback
AUTH0_CLIENT_SECRET Production auth Secret Deployment secret injection Auth0 callback
AUTH0_ORGANIZATION_ID Production auth Optional org routing Deployment env injection Auth0 callback
AUTH0_SECRET Production auth Secret Deployment secret injection Session cookie encryption
AUTH0_BASE_URL Production auth App origin Deployment env injection Auth0 callback
AUTH0_ISSUER_BASE_URL Production auth Tenant issuer origin Deployment env injection Auth0 callback
AUTH0_MANAGEMENT_CLIENT_ID Production auth management Server-only identifier Deployment env injection Auth0 management helper
AUTH0_MANAGEMENT_CLIENT_SECRET Production auth management Secret Deployment secret injection Auth0 management helper
AUTH0_MANAGEMENT_AUDIENCE Production auth management API audience Deployment env injection Auth0 management helper

Vertex AI training, verification, and runtime scoring variables

documents/vertex_ai_verification.md and backend/src/usecase/credit_score_runtime.py

Variable Phase Scope Ownership Used by
PROJECT_ID Training and runtime GCP project identifier Deployment env Compose helpers, Vertex jobs, runtime scoring
REGION Training and runtime Vertex location Deployment env Compose helpers, runtime scoring
DATA_URI Training Input dataset location Deployment env Training container
MODEL_OUTPUT_DIR Training Artifact output location Deployment env Training container
PREDICTION_CONTAINER_IMAGE Deployment Serving image Deployment env Vertex deployment job
TRAIN_IMAGE_URI Build and run Training image tag Deployment env Compose build/push and Vertex jobs
TRAIN_MACHINE_TYPE Training Worker shape Deployment env Vertex training job
TRAIN_MACHINE_TYPE_FALLBACK Training Retry worker shape Deployment env Pipeline retry logic
TRAIN_MACHINE_TYPES Training Ordered retry list Deployment env Pipeline retry logic
DEPLOY_MACHINE_TYPE Deployment Endpoint worker shape Deployment env Vertex deploy step
TARGET_COLUMN Training Label override Deployment env Training script
TARGET_SCORE_THRESHOLD Training Continuous label binarization Deployment env Training script
MAX_TRAIN_ROWS Training Row sampling cap Deployment env Training script
FEATURE_COLUMNS_LIMIT Training Feature cap Deployment env Training script
CREDIT_SCORE_PROVIDER Runtime scoring Provider selector Runtime env Applicant credit-score flow
VERTEX_CREDIT_ENDPOINT_NAME Runtime scoring Endpoint resource name Runtime env backend/src/usecase/credit_score_runtime.py
VERTEX_CREDIT_ENDPOINT_ID Runtime scoring Endpoint id fallback Runtime env backend/src/usecase/credit_score_runtime.py
CREDIT_SCORE_ALLOW_HEURISTIC_FALLBACK Runtime scoring Fail-open switch Runtime env Runtime verification path
VERIFY_STRICT Verification Strict readiness mode Runtime env Verification scripts
WAIT_FOR_TRAFFIC_SPLIT_SECONDS Verification Readiness wait Runtime env Verification scripts
VERIFY_ALLOW_SHAPE_MISMATCH Verification Serving-contract policy Runtime env Verification scripts
VERIFY_PREDICT Verification Smoke-predict toggle Runtime env Verification scripts
BUILD_BEFORE_RUN Orchestration Build toggle Runtime env Pipeline runner
PUSH_IMAGE_BEFORE_RUN Orchestration Push toggle Runtime env Pipeline runner
RUN_DEPLOY_STEP Orchestration Deploy toggle Runtime env Pipeline runner
RUN_PREDICT_STEP Orchestration Prediction toggle Runtime env Pipeline runner
FAIL_ON_PREDICT_ERROR Orchestration Fail-hard toggle Runtime env Pipeline runner
IMAGE_IMPORT_CHECK Verification Image sanity check Runtime env Fast credit check
FAST_CHECK_RESULT Verification Final status output Runtime env Fast credit check
FAST_CHECK_REPORT_PATH Verification Report output path Runtime env Fast credit check
GCP_CREDENTIALS_PATH Registry/auth helper Service account key path Secret file or env Registry auth helper
SERVICE_ACCOUNT_EMAIL Registry/auth helper Impersonation target Deployment env Registry auth helper

Observability variables

frontend/.env.local.example

Variable Scope Visibility Notes
NEXT_PUBLIC_SENTRY_DSN Browser Sentry Public Client-side error reporting
SENTRY_DSN Server Sentry Server-side Backend and server-rendered telemetry
NEXT_PUBLIC_SENTRY_ENVIRONMENT Browser Sentry Public Browser environment label
SENTRY_ENVIRONMENT Server Sentry Server-side Server environment label
NEXT_PUBLIC_SENTRY_TRACES_SAMPLE_RATE Browser tracing Public Sampling control
SENTRY_TRACES_SAMPLE_RATE Server tracing Server-side Sampling control
NEXT_PUBLIC_SENTRY_REPLAYS_SESSION_SAMPLE_RATE Browser replay Public Session replay sampling
NEXT_PUBLIC_SENTRY_REPLAYS_ON_ERROR_SAMPLE_RATE Browser replay Public Error-triggered replay
SENTRY_AUTH_TOKEN Release tooling Secret Source map and release upload
SENTRY_ORG Release tooling Secret Release upload
SENTRY_PROJECT Release tooling Secret Release upload
NEXT_PUBLIC_AMPLITUDE_API_KEY Analytics Public Browser analytics key
OTEL_SERVICE_NAME Tracing Deployment env Default is lita-frontend
OTEL_EXPORTER_OTLP_ENDPOINT Tracing Deployment env OTLP collector target
OTEL_EXPORTER_OTLP_HEADERS Tracing Deployment env or secret store Export headers

Secret Ownership and Storage

Ownership boundary Variables and files Storage location
Frontend public bundle NEXT_PUBLIC_*, NEXT_PUBLIC_AMPLITUDE_API_KEY, NEXT_PUBLIC_SENTRY_*, NEXT_PUBLIC_AUTH0_ORGANIZATION_ID Image build args or runtime env
Frontend server-only auth AUTH0_CLIENT_SECRET, AUTH0_SECRET, AUTH0_MANAGEMENT_CLIENT_SECRET Deployment secret store
Auth0 identity config AUTH0_DOMAIN, AUTH0_ISSUER_BASE_URL, AUTH0_CLIENT_ID, AUTH0_ORGANIZATION_ID, AUTH0_MANAGEMENT_AUDIENCE Deployment env files
Vertex runtime scoring CREDIT_SCORE_PROVIDER, PROJECT_ID, REGION, VERTEX_CREDIT_ENDPOINT_NAME, VERTEX_CREDIT_ENDPOINT_ID, CREDIT_SCORE_ALLOW_HEURISTIC_FALLBACK Deployment .env files
Vertex training and verification DATA_URI, MODEL_OUTPUT_DIR, TRAIN_IMAGE_URI, TRAIN_MACHINE_TYPE, TRAIN_MACHINE_TYPES, DEPLOY_MACHINE_TYPE, TARGET_COLUMN, TARGET_SCORE_THRESHOLD, MAX_TRAIN_ROWS, FEATURE_COLUMNS_LIMIT Deployment .env files
CI/CD and rollout GCP_SA_KEY, GCP_PROJECT_ID, COMPUTE_ENGINE_IP, COMPUTE_ENGINE_USER, SSH_PRIVATE_KEY GitHub Actions secrets
GCP helper credentials GCP_CREDENTIALS_PATH, SERVICE_ACCOUNT_EMAIL Local secret file or env injection
Application data storage APPLICATIONS_DATA_GCS_BUCKET, APPLICATIONS_DATA_GCS_OBJECT Deployment env files

Runtime Configuration Surfaces

Frontend backend URL resolution

[!NOTE] AUTH0_CLIENT_ID and AUTH0_DOMAIN are identifiers, not secrets. The secret-bearing Auth0 values in this repo are AUTH0_CLIENT_SECRET, AUTH0_SECRET, and AUTH0_MANAGEMENT_CLIENT_SECRET.

frontend/lib/utils.ts

getConfiguredBackendApiUrl

Method Description
getConfiguredBackendApiUrl Resolves the backend origin from NEXT_PUBLIC_BACKEND_API_URL, or falls back to the browser origin when the configured host does not match the current local network context.

This helper is the frontend’s main deployment-aware routing surface. It prefers NEXT_PUBLIC_BACKEND_API_URL on both server and client, but on the browser it can replace a local-network URL with the current window.location.origin when the configured host does not match the browser host. That behavior is what keeps local and production deployments aligned without hard-coding the origin into the app shell.

Auth0 edge auth and management client

frontend/lib/auth0-management.ts, frontend/app/api/auth/auth0/[auth0]/route.ts, frontend/lib/auth-constants.ts

Configuration constants

Constant Value Purpose
AUTH0_SCOPE openid profile email OAuth scope used by the callback flow
AUTH0_STATE_COOKIE_NAME auth0_oauth_state OAuth state validation cookie
AUTH0_SESSION_COOKIE_NAME auth0_session Session cookie name

Auth0 user model

frontend/app/api/auth/auth0/[auth0]/route.ts

Property Type Description
id number Local user identifier assigned by the callback helper
username string Resolved from preferred_username, nickname, name, or email prefix
email string Normalized email address
role string Set to admin for the hard-coded admin email list, otherwise applicant
applicationId `number \ null`

Public methods

frontend/lib/auth0-management.ts

Method Description
findAuth0UserByEmail Calls the Auth0 Management API to look up a user by normalized email address.
createAuth0User Creates a password-user in the Username-Password-Authentication connection with the supplied identity fields.
getAuth0UserRoles Fetches role names for a user and normalizes them to lowercase strings.

Auth0 Management API calls

Find User By Email
{
    "title": "Find Auth0 User By Email",
    "description": "Looks up Auth0 users by normalized email address",
    "method": "GET",
    "baseUrl": "<AUTH0_MANAGEMENT_AUDIENCE>",
    "endpoint": "/api/v2/users-by-email",
    "headers": [
        {
            "key": "Authorization",
            "value": "Bearer <management access token>",
            "required": true
        }
    ],
    "queryParams": [
        {
            "key": "email",
            "required": true,
            "description": "Lowercased email address"
        }
    ],
    "pathParams": [],
    "bodyType": "none",
    "requestBody": "",
    "formData": [],
    "rawBody": "",
    "responses": {
        "200": {
            "description": "Matching users",
            "body": "[\n    {\n        \"user_id\": \"auth0|abc123\"\n    }\n]"
        }
    }
}
Create Auth0 User
{
    "title": "Create Auth0 User",
    "description": "Creates a password user in Auth0 and returns the Auth0 user identifier",
    "method": "POST",
    "baseUrl": "<AUTH0_MANAGEMENT_AUDIENCE>",
    "endpoint": "/api/v2/users",
    "headers": [
        {
            "key": "Authorization",
            "value": "Bearer <management access token>",
            "required": true
        },
        {
            "key": "Content-Type",
            "value": "application/json",
            "required": true
        }
    ],
    "queryParams": [],
    "pathParams": [],
    "bodyType": "json",
    "requestBody": "{\n    \"connection\": \"Username-Password-Authentication\",\n    \"email\": \"[email protected]\",\n    \"password\": \"DemoPassword123!\",\n    \"username\": \"dwdemo\",\n    \"name\": \"Demo Applicant\",\n    \"given_name\": \"Demo\",\n    \"family_name\": \"Applicant\",\n    \"verify_email\": false,\n    \"email_verified\": true\n}",
    "formData": [],
    "rawBody": "",
    "responses": {
        "200": {
            "description": "Created user",
            "body": "{\n    \"user_id\": \"auth0|abc123\"\n}"
        }
    }
}
Get Auth0 User Roles
{
    "title": "Get Auth0 User Roles",
    "description": "Returns role names assigned to an Auth0 user",
    "method": "GET",
    "baseUrl": "<AUTH0_MANAGEMENT_AUDIENCE>",
    "endpoint": "/api/v2/users/{userId}/roles",
    "headers": [
        {
            "key": "Authorization",
            "value": "Bearer <management access token>",
            "required": true
        }
    ],
    "queryParams": [],
    "pathParams": [
        {
            "key": "userId",
            "required": true,
            "description": "Auth0 user identifier"
        }
    ],
    "bodyType": "none",
    "requestBody": "",
    "formData": [],
    "rawBody": "",
    "responses": {
        "200": {
            "description": "Assigned roles",
            "body": "[\n    {\n        \"name\": \"admin\"\n    }\n]"
        }
    }
}

The management helper normalizes AUTH0_DOMAIN by stripping any protocol and trailing slash, and it falls back to AUTH0_ISSUER_BASE_URL when AUTH0_DOMAIN is not present. AUTH0_MANAGEMENT_AUDIENCE is also derived from the domain when it is omitted, which keeps the management API base aligned with the tenant.

Vertex AI runtime scoring

backend/src/usecase/credit_score_runtime.py and documents/vertex_ai_verification.md

_resolve_endpoint_name accepts either VERTEX_CREDIT_ENDPOINT_NAME or VERTEX_CREDIT_ENDPOINT_ID. If the value already starts with projects/, it is used as-is; otherwise it is expanded into the full Vertex resource name using PROJECT_ID and REGION.

Runtime scoring flow

sequenceDiagram
    participant Operator as Operator
    participant Script as verify_runtime_credit_scoring.sh
    participant Env as deployment env file
    participant Backend as credit score runtime helper
    participant Vertex as Vertex AI endpoint

    Operator->>Script: run verification
    Script->>Env: load PROJECT_ID REGION and endpoint vars
    Script->>Backend: resolve endpoint name
    Backend-->>Script: projects project locations region endpoints id
    Script->>Vertex: predict
    Vertex-->>Script: prediction response or serving error

The verifier first validates the env file, then tries the Python SDK and falls back to gcloud ai endpoints predict when the SDK is unavailable. It can auto-select a deployed endpoint with traffic, wait for a traffic split, or fail closed when VERIFY_STRICT=1 or CREDIT_SCORE_ALLOW_HEURISTIC_FALLBACK=false are set.

Deployment-level state and cache surfaces

backend/src/interface/saltedge_endpoints.py

Variable Type Description
_customer_cache Dict[str, Any] In-process Salt Edge customer cache with cached response payload and expiry
_customer_cache_lock asyncio.Lock Serializes cache refreshes across concurrent requests
CUSTOMER_CACHE_TTL_SECONDS int Cache lifetime, set to 60 seconds

Cache behavior

  • Requests to the Salt Edge customer listing path check _customer_cache before calling the external client.
  • Cache hits return the stored data value without re-fetching.
  • Cache misses acquire _customer_cache_lock, call the external Salt Edge client, and refresh the cache with a new expiry timestamp.
  • The cache is process-local and keyed as a single shared customer list, not per customer.
sequenceDiagram
    participant Client as API consumer
    participant Api as Salt Edge customer endpoint
    participant Cache as in process cache
    participant SaltEdge as Salt Edge client

    Client->>Api: request customer list
    Api->>Cache: read cached data
    alt cache hit
        Cache-->>Api: cached payload
    else cache miss
        Api->>Cache: acquire lock
        Api->>SaltEdge: fetch customer list
        SaltEdge-->>Api: response data
        Api->>Cache: store data and expiry
    end
    Api-->>Client: response data

Observability and Logging

The frontend observability contract is entirely env-driven. Sentry is configured twice, once for browser use via NEXT_PUBLIC_* and once for server-side reporting via SENTRY_*. Amplitude is exposed to the browser through NEXT_PUBLIC_AMPLITUDE_API_KEY, and OpenTelemetry is configured through OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT, and OTEL_EXPORTER_OTLP_HEADERS.

The backend uses standard Python logging with module-level loggers in the Salt Edge endpoint module and the research service. Operational messages include cache hits and misses, Salt Edge API errors, and research job failures. Those log lines are the primary runtime signal for deployment issues in the source shown here.

Configuration Precedence and Error Handling

  • NEXT_PUBLIC_BACKEND_API_URL wins for frontend backend routing; if it is absent, getConfiguredBackendApiUrl falls back to the browser origin when available.
  • AUTH0_DOMAIN wins over AUTH0_ISSUER_BASE_URL in the Auth0 helper.
  • VERTEX_CREDIT_ENDPOINT_NAME wins over VERTEX_CREDIT_ENDPOINT_ID.
  • AUTH0_ORGANIZATION_ID wins over AUTH0_ORGANIZATION, and the client-side fallback uses NEXT_PUBLIC_AUTH0_ORGANIZATION_ID or NEXT_PUBLIC_AUTH0_ORGANIZATION.
  • Missing Auth0 configuration returns a configuration failure before token exchange.
  • The Auth0 callback path redirects back to /login with explicit error markers for invalid state, token exchange failure, and missing tokens.
  • The Vertex runtime helper raises an environment error when neither endpoint env var is present.
  • The Salt Edge integration path returns HTTP 503 when the client cannot be configured.

Key Classes Reference

[!NOTE] AUTH0_STATE_COOKIE_NAME and AUTH0_SESSION_COOKIE_NAME are fixed constants in frontend/lib/auth-constants.ts. They are configuration surfaces in code, but not env-file toggles.

Class Responsibility
frontend/lib/utils.ts Resolves the frontend backend URL from environment and browser origin.
frontend/lib/auth0-management.ts Auth0 Management API wrapper for lookup, creation, and role reads.
frontend/app/api/auth/auth0/[auth0]/route.ts Auth0 callback and local user bootstrap flow.
frontend/lib/auth-constants.ts Fixed cookie names used by Auth0 state and session handling.
backend/src/usecase/credit_score_runtime.py Resolves Vertex endpoint resource names from runtime env vars.
backend/src/interface/saltedge_endpoints.py Salt Edge integration endpoints plus in-process customer cache.
frontend/.env.local.example Frontend runtime, auth, observability, and client-visible env contract.
documents/deployment.md Production deployment env expectations and rollout settings.
documents/vertex_ai_verification.md Vertex training, deployment, and runtime verification variable contract.