Credit Scoring and Pipeline Domain Feature Engineering, Canonical Schema, and Lineage Controls

Overview

This domain turns housing application data, consented open-banking data, bureau records, and macro context into a single canonical credit input that downstream scoring code can consume consistently. The key goal is not just scoring, but traceability: every model-facing field is mapped back to a canonical source path, raw payloads are preserved for audit, and feature derivation checks can catch inconsistencies before a score is emitted.

The scoring path is split into three tightly coupled concerns. credit_canonical_schema.py defines the canonical data containers, credit_adapters.py normalizes provider payloads into those containers, and credit_variable_map.py defines the authoritative lineage table that links canonical paths to model features. Feature builders in open_banking_features.py and credit_model.py then turn canonical data into model-ready columns, while lita_global_score_converter.py and the runtime scorer convert calibrated probabilities into Lita legacy scores, global scores, and band labels.

Architecture Overview

flowchart TB
    subgraph Sources[Input Sources]
        App[Application data]
        Bureau[Bureau payloads]
        Banking[Open banking payloads]
        Macro[Macro context]
    end

    subgraph Canonicalization[Canonicalization Layer]
        Adapters[credit_adapters]
        Schema[credit_canonical_schema]
    end

    subgraph Lineage[Lineage and Validation Layer]
        Map[credit_variable_map]
        Tests[test_variable_derivation]
        Validation[credit_pipeline_validation]
    end

    subgraph Features[Feature Engineering Layer]
        OB[open_banking_features]
        Model[credit_model]
    end

    subgraph Calibration[Calibration Layer]
        Cal[lita_global_score_converter]
        Resp[CreditScoreResponse]
    end

    App --> Adapters
    Bureau --> Adapters
    Banking --> Adapters
    Macro --> Schema
    Adapters --> Schema
    Schema --> Map
    Schema --> Validation
    Schema --> OB
    Schema --> Model
    OB --> Cal
    Model --> Cal
    Map --> Tests
    Validation --> Resp
    Cal --> Resp

Canonical Data Models

Canonical Bureau Data

backend/src/ml/credit_canonical_schema.py

CanonicalBureauData is the provider-agnostic bureau container. It normalizes TransUnion rows, Equifax payloads, manual uploads, and other bureau sources into a single structure before feature engineering runs.

Property Type Description
provider str Bureau source name such as transunion, equifax, manual_upload, or global.
score_value `float \ None` Raw bureau score in the provider scale.
score_scale_min float Lower bound of the provider score range.
score_scale_max float Upper bound of the provider score range.
score_provider str Provider-specific score family such as vantage_4 or equifax_risk.
utilization_ratio `float \ None` Overall revolving utilization.
delinquencies_24m int Count of delinquency events in the last 24 months.
worst_dpd_24m int Worst days-past-due bucket in the last 24 months.
credit_age_months `float \ None` Age of the newest trade in months.
credit_age_months_oldest `float \ None` Age of the oldest trade in months.
inquiries_12m int Inquiries in the last 12 months.
new_accounts_12m int New accounts in the last 12 months.
collections_flag bool Collections presence flag.
collections_count int Collections count.
default_flag bool Default indicator.
report_recency_days float Days since the bureau report was pulled.
parse_confidence float OCR or parser confidence.
fraud_risk_flag bool Fraud risk indicator.
aml_risk_flag bool AML risk indicator.
estimated_income_score `float \ None` Bureau-estimated income proxy.
revolving_util_patterns dict[str, float] Trended revolving utilization pattern values.
bankcard_util_patterns dict[str, float] Trended bankcard utilization pattern values.
trade_age_months dict[str, float] Trade-age buckets keyed by trade type.
loan_balances dict[str, float] Loan balance buckets keyed by loan type.
raw dict[str, Any] Original bureau payload retained for audit.

Canonical Open Banking Data

backend/src/ml/credit_canonical_schema.py

CanonicalOpenBankingData stores the normalized open-banking view used by downstream scoring and training.

Property Type Description
provider str Provider source name such as plaid, saltedge, finverse, manual, or mixed.
institutions list[str] Linked financial institutions.
accounts_count int Count of linked accounts.
days_covered int Calendar days spanned by the transaction history.
transactions_90d list[CanonicalTransaction] Canonical 90-day transaction list.
balance_snapshots_usd list[float] USD-normalized balance readings.
raw dict[str, Any] Original open-banking payload retained for audit.

Canonical Credit Input

backend/src/ml/credit_canonical_schema.py

CanonicalCreditInput is the canonical intermediate object that holds application-level facts together with optional bureau and banking providers.

Property Type Description
application_id str Application identifier used across the pipeline.
bureau `CanonicalBureauData \ None` Optional bureau payload after normalization.
banking `CanonicalOpenBankingData \ None` Optional open-banking payload after normalization.
income_stated_usd float Applicant-stated income in USD.
savings_usd float Applicant-stated savings in USD.
budget_usd float Applicant-stated budget in USD.
employment_status str Applicant employment status.
home_country str Applicant home country.
host_country str Host country used for corridor context.
product_type str Product type, defaulting to rental.
kyc_passed bool KYC completion flag.
host_unemployment_rate float Macro unemployment context.
host_inflation_rate float Macro inflation context.
host_interest_rate_proxy float Macro interest-rate proxy.
data_completeness DataCompletenessReport Populated after validation with completeness and gating state.

Scoring Output Contract

backend/src/interface/schemas.py

CreditScoreResponse is the calibrated output shape consumed by the runtime scoring layer.

Property Type Description
application_id int Application identifier.
score int Legacy 300–850 score.
score_band Literal["A", "B", "C", "D", "E"] Legacy score band.
provider str Score provider name.
model_version str Model or policy version string.
featureset_version str Featureset version string.
calibration_version str Calibration registry version.
confidence float Confidence score from 0 to 1.
data_coverage Dict[str, Any] Coverage summary captured at scoring time.
reason_codes List[str] Reason codes emitted by the scoring flow.
breakdown Dict[str, float] Score component breakdown.
lita_global_score int Global Lita score on the 0–1000 scale.
global_band Literal["A", "B", "C", "D", "E"] Global score band.
risk_flags List[Dict[str, str]] Structured risk flags.
target_scale Optional[str] Requested target scale, if any.
target_score Optional[int] Score projected onto the target scale.
target_confidence Optional[float] Confidence for the target-scale projection.

Feature Engineering

Bureau Feature Contract and Derived Columns

backend/src/ml/credit_model.py

The bureau training path uses raw TransUnion-style columns, renames them into canonical names via COLUMN_RENAMES, and then derives aggregate features in build_features. FEATURE_COLUMNS is the full bureau-side model input, and ALL_FEATURE_COLUMNS appends open-banking features for the unified model.

Constant Type Role
COLUMN_RENAMES dict[str, str] Renames raw bureau fields to canonical model-facing names.
USE_COLUMNS list[str] Raw input columns consumed by the training path.
BASE_FEATURE_COLUMNS list[str] Canonical bureau columns excluding the target.
ENGINEERED_COLUMNS list[str] Bureau-derived aggregate features computed in build_features.
FEATURE_COLUMNS list[str] Bureau base columns plus engineered columns.
OPEN_BANKING_FEATURE_COLUMNS list[str] Shared open-banking feature contract used by training and runtime.
ALL_FEATURE_COLUMNS list[str] Unified feature contract: bureau features plus open-banking features.
OPEN_BANKING_LEAKAGE_TOKENS tuple[str, ...] Tokens excluded from open-banking inputs: target, label, outcome.

Derived bureau features computed by build_features

Derived column How it is built
balance_total Sum of LOAN_BALANCE_COLS.
num_tradelines Sum of REVOLVING_TRADE_COLS and BANKCARD_TRADE_COLS.
balance_per_trade balance_total / (trade_counts + 1).
util_bankcard_weighted Mean of BANKCARD_PATTERN_COLS.
util_revolving_weighted Mean of REVOLVING_PATTERN_COLS.
util_total Sum of weighted bankcard and revolving utilization.
util_total_sq Square of util_total.
util_bank_x_rev Product of bankcard and revolving utilization.
average_utilization Mean of the two weighted utilization features.
utilization_volatility Standard deviation of the two weighted utilization features.
loan_ratio balance_total / (income + 1).
debt_burden_score revolving_balance_proxy / (income + 1).
is_thin_file 1 when num_tradelines < 5, otherwise 0.
credit_history_age_mean Mean of TRADE_AGE_COLS.
credit_history_age_max Max of TRADE_AGE_COLS.
credit_history_age_min Min of TRADE_AGE_COLS.
credit_history_age_span credit_history_age_max - credit_history_age_min.

Open Banking Feature Engineering

backend/src/ml/open_banking_features.py

build_open_banking_feature_vector assembles the open-banking feature row by combining income, balance, spending, stability, and country-risk feature dictionaries into one DataFrame row. build_balance_features is the visible balance-side builder and standardizes balance snapshots into a small set of liquidity metrics.

Method Description
build_balance_features Builds balance-derived liquidity features from historical_balances, bounce_count, inflow, and outflow.
build_open_banking_feature_vector Merges the open-banking feature families into a single feature row and appends extra_features when provided.

Balance feature outputs from build_balance_features

Output column Description
avg_daily_balance Mean of historical_balances.
min_balance_30d Minimum balance over the tail 30 readings.
overdraft_count Count of balances below zero.
bounce_count Sum of bounce_count.
cash_buffer_ratio avg_daily_balance / (outflow + 1) when outflow exists.
net_cashflow_month inflow - outflow.
surplus_ratio net_cashflow_month / (inflow + 1).

Open banking feature contract

The shared open-banking feature list in credit_model.py includes income, affordability, account-linking, transaction coverage, salary behavior, spending, obligations, balance, and macro-risk features. The contract is intentionally shared between training-time feature assembly and runtime feature extraction so the same columns are seen by the model in both paths.

Feature Builder Alignment

backend/src/usecase/credit_score_runtime.py and backend/src/ml/credit_model.py

Runtime feature construction and model-training features are aligned through the same naming contract.

  • credit_model.py uses FEATURE_COLUMNS for bureau-derived features and OPEN_BANKING_FEATURE_COLUMNS for open-banking features.
  • _prepare_open_banking_features fills missing open-banking columns with 0.0 and strips any column whose name contains a leakage token.
  • build_features and build_open_banking_feature_vector both normalize into flat model-ready shapes, with ALL_FEATURE_COLUMNS acting as the unified model input set.

Adapters and Source Normalization

Credit Adapters

backend/src/ml/credit_adapters.py

The adapter layer is the only place that understands provider-specific payload shapes. It converts bureau and open-banking sources into canonical dataclasses and keeps the original payload in raw for auditability.

Method Description
BureauAdapter.from_transunion_row() Normalizes a renamed TransUnion batch row into CanonicalBureauData.
BureauAdapter.from_equifax_payload() Normalizes an Equifax instant-intake payload into CanonicalBureauData.
BureauAdapter.from_manual_report() Normalizes an OCR-extracted or manually uploaded bureau report into CanonicalBureauData.
OpenBankingAdapter.from_connected_accounts() Normalizes a generic connected-accounts structure into CanonicalOpenBankingData.
OpenBankingAdapter.from_plaid() Normalizes a Plaid /accounts and /transactions envelope into canonical open-banking data.
OpenBankingAdapter.from_saltedge() Normalizes a Salt Edge payload into canonical open-banking data.

Adapter responsibilities visible in the code

  • CanonicalBureauData keeps provider scores, delinquency history, utilization patterns, trade-age buckets, loan balances, and raw source payload.
  • CanonicalOpenBankingData keeps institutions, account counts, coverage length, transaction history, balance snapshots, and raw source payload.
  • _BUREAU_KEYS = {"bureau_report", "manual_credit_report", "bureau"} marks embedded bureau payloads inside connected-account structures so bureau data is not treated as banking data.
  • _DEFAULT_FX_TO_USD provides the FX table used to convert monetary values to USD during open-banking normalization.

Provider normalization matrix

Source path Canonical target Key fields preserved
from_transunion_row() CanonicalBureauData Score, recency, trended utilization, trade ages, loan balances, raw row.
from_equifax_payload() CanonicalBureauData Risk score scale, tradeline delinquency counts, utilization, recency, parser confidence, raw payload.
from_manual_report() CanonicalBureauData Score, score scale, provider name, recency, confidence, risk flags, raw payload.
from_connected_accounts() CanonicalOpenBankingData Linked institutions, accounts, transactions, balances, and source payload.

Lineage and Validation Controls

Variable Map and Feature Lineage

backend/src/ml/credit_variable_map.py

VARIABLE_MAP is the authoritative lineage table. Each row is a 4-tuple:

Tuple item Type Meaning
canonical_path str Dotted path into CanonicalCreditInput or nested provider data.
feature_column str Feature name used by the model.
source_type str One of bureau, banking, application, macro.
required bool Whether the feature must be present for a valid score.

Derived lineage sets

Constant Type Responsibility
REQUIRED_FEATURE_COLUMNS frozenset[str] All feature columns marked required.
BANKING_DERIVED_FEATURES frozenset[str] Features sourced from banking lineage.
BUREAU_DERIVED_FEATURES frozenset[str] Features sourced from bureau lineage.
ALL_MAPPED_FEATURES frozenset[str] All feature columns declared in VARIABLE_MAP.
CANONICAL_SOURCES frozenset[str] All canonical source paths used by the map.

Source families covered by VARIABLE_MAP

  • Application: income_stated_usd, savings_usd, budget_usd, employment_status, kyc_passed, product_type
  • Macro: host_unemployment_rate, host_inflation_rate, host_interest_rate_proxy
  • Bureau: score, utilization, delinquency, age, inquiries, new accounts, collections, default, income estimate, trended utilization buckets, trade-age buckets, loan balances
  • Banking: institution counts, account counts, days covered, coverage score, salary and income behavior, spending, obligations, affordability, balance and overdraft behavior

Validation Helpers

backend/src/ml/credit_variable_map.py

Method Description
validate_feature_coverage() Checks whether the built feature dict contains the mapped columns, with required_only controlling whether optional columns are included.
detect_derivation_mismatches() Returns human-readable derivation issues for a built feature dict.
list_unmapped_model_columns() Surfaces model columns that lack lineage coverage.
list_mapped_but_unused_columns() Surfaces lineage entries that are no longer used by the model.

Mismatch rules enforced by detect_derivation_mismatches

Rule family Condition checked Expected outcome
Salary consistency salary_detected=True and salary_payer_count_90d=0 Flags missing payer-set derivation.
Salary consistency salary_detected=True and salary_payment_count_90d=0 Flags missing salary payment iteration.
Salary consistency salary_detected=True and salary_regularity_score=0.0 Flags missing regularity computation.
Salary consistency salary_detected=False and salary_payer_count_90d>0 Flags stale payer leakage.
Non-salary income Banking available, no salary, and non_salary_income_share=0.0 Flags missing non-salary income derivation.
Non-salary income salary_detected=False and income_sources_count>1 Flags inconsistent source counting.
Non-salary income income_value=0.0 with banking data present Flags missing income in the feature vector.
Bureau consistency Bureau available and bureau_available_flag=0.0 Flags adapter-to-feature propagation failure.
Bureau consistency bureau_available_flag=1 and bureau_score_value=0.0 Flags missing bureau score mapping.
Bureau consistency No bureau input and bureau_score_value>0.0 Flags impossible bureau leakage.
Affordability affordability_ratio differs from budget_value / income_value by more than 0.02 Flags possible FX normalization or derivation mismatch.
Affordability income_value>0 and budget_value=0.0 Flags missing budget propagation.
Coverage Banking available and coverage_score=0.0 Flags missing coverage-score computation.
Coverage days_covered>0 and coverage_score=0.0 Flags missing translation from history length to coverage.
Balance consistency Banking available and avg_balance_90d=0.0 Flags missing balance snapshot population.
Balance consistency min_balance_30d > avg_balance_30d Flags window mismatch or sign error.

Coverage Rules and Required Columns

backend/src/ml/credit_variable_map.py and backend/tests/ml/test_variable_derivation.py

The tests make the required-column behavior explicit:

  • validate_feature_coverage(..., required_only=True) checks only REQUIRED_FEATURE_COLUMNS.
  • validate_feature_coverage(..., required_only=False) also checks optional bureau columns.
  • 0 is treated as a valid value.
  • None is treated as missing.
  • The returned missing-column list is sorted.

The structural tests also enforce that:

  • every VARIABLE_MAP row is a 4-tuple,
  • every source_type is one of bureau, banking, application, macro,
  • REQUIRED_FEATURE_COLUMNS is a subset of ALL_MAPPED_FEATURES,
  • banking and bureau feature sets do not overlap,
  • bureau-derived columns are not marked required.

Completeness and Gating State

Pipeline Completeness Controls

backend/src/ml/credit_pipeline_validation.py

This module consumes CanonicalCreditInput and produces a DataCompletenessReport that drives gating before scoring.

Constant Type Responsibility
MIN_DAYS_COVERED int Minimum acceptable open-banking history length.
BUREAU_MAX_AGE_DAYS float Maximum acceptable bureau recency age.
MIN_COMPLETENESS_SCORE_TO_PASS float Minimum completeness score needed to pass.
_BUREAU_REQUIRED_PRODUCTS frozenset[str] Product types that require bureau data.
_DECLINE_FLAGS frozenset[str] Flags that cause a decline outcome.
_REFER_FLAGS frozenset[str] Flags that cause a refer outcome.

Gating priority

  1. DECLINE for KYC_NOT_PASSED, FRAUD_RISK_DETECTED, AML_RISK_DETECTED
  2. REFER for BUREAU_REQUIRED_BUT_ABSENT, NO_BANKING_DATA, INSUFFICIENT_HISTORY
  3. PASS for all other cases

Completeness fields populated in the flow

  • kyc_passed
  • banking_available
  • days_covered_ok
  • salary_detected
  • income_verified
  • bureau_available
  • bureau_recency_ok
  • flags
  • completeness_score
  • gate_result

Score Calibration and Global Bands

Lita Global Score Conversion

backend/src/ml/lita_global_score_converter.py

The calibration layer transforms raw or adjusted probability-of-default values into the score bands used by the rest of the system.

Method Description
apply_coverage_honesty_policy() Applies coverage-sensitive policy adjustments before the score is finalized.
calibrate_pd() Calibrates PD using the corridor registry, product type, and corridor cluster.
confidence_score() Computes the confidence value associated with the calibrated output.
pd_band() Maps probability values to a PD band.
pd_to_global_score() Converts calibrated PD into the Lita global score scale.
pd_to_legacy_score() Converts calibrated PD into the legacy 300–850 score scale.

Calibration flow used by the runtime scorer

backend/src/usecase/credit_score_runtime.py

The runtime scorer passes product_type and corridor_cluster into calibrate_pd, then applies coverage policy before computing both score scales. The visible flow is:

  • calibrate_pd(risk.pd_p50_12m, registry=_CORRIDOR_CALIBRATION_REGISTRY, product_type=..., corridor_cluster=...)
  • apply_coverage_honesty_policy(pd=calibrated_pd, features=features, product_type=...)
  • pd_to_legacy_score(policy_pd)
  • pd_to_global_score(adjusted_pd)
  • legacy and global bands are derived after score conversion

Output fields that carry calibration state

CreditScoreResponse preserves both scales and the band labels:

  • score and score_band
  • lita_global_score and global_band
  • calibration_version
  • confidence
  • target_scale, target_score, target_confidence

Feature and Data Flow

Canonicalization and Feature Assembly

sequenceDiagram
    participant Src as Provider payload
    participant Adp as credit_adapters
    participant Sch as credit_canonical_schema
    participant Feat as feature builders
    participant Map as credit_variable_map

    Src->>Adp: from_transunion_row or from_equifax_payload or from_manual_report
    Adp->>Sch: CanonicalBureauData
    Src->>Adp: from_connected_accounts
    Adp->>Sch: CanonicalOpenBankingData
    Sch->>Feat: build_features or build_open_banking_feature_vector
    Feat-->>Sch: model-ready feature row
    Sch->>Map: validate_feature_coverage
    Map-->>Sch: missing columns
    Sch->>Map: detect_derivation_mismatches
    Map-->>Sch: issue list

Calibration to Lita scores

sequenceDiagram
    participant Risk as raw PD output
    participant Cal as lita_global_score_converter
    participant Run as runtime scorer
    participant Resp as CreditScoreResponse

    Risk->>Cal: calibrate_pd
    Cal-->>Run: calibrated PD and calibration version
    Run->>Cal: apply_coverage_honesty_policy
    Cal-->>Run: policy adjusted PD and policy adjustments
    Run->>Cal: pd_to_legacy_score
    Run->>Cal: pd_to_global_score
    Run->>Cal: pd_band
    Run->>Resp: populate score, lita_global_score, bands

State Management

Canonical pipeline state progression

The domain uses an explicit progression rather than ad hoc field mutation:

  1. Provider payloads enter the adapter layer.
  2. Adapters produce CanonicalBureauData and CanonicalOpenBankingData.
  3. CanonicalCreditInput binds bureau, banking, application, and macro context together.
  4. credit_pipeline_validation.py populates data_completeness with flags, completeness score, and gate result.
  5. Feature builders create model-ready columns.
  6. credit_variable_map.py checks coverage and derivation consistency.
  7. The calibration layer turns probability outputs into score scales and bands.
  8. The runtime scorer persists an audit artifact that includes feature_hash, calibration_version, reason_codes, and adjustments_applied.

Integration Points

  • backend/src/usecase/credit_score_runtime.py consumes calibrated outputs and writes scoring artifacts with lineage metadata.
  • backend/src/interface/schemas.py defines the response contract that carries both legacy and global scores.
  • backend/src/ml/credit_pipeline_validation.py consumes CanonicalCreditInput and sets the completeness gate state.
  • backend/tests/ml/test_pipeline_validation.py and backend/tests/ml/test_variable_derivation.py exercise the canonicalization and lineage rules.
  • backend/src/usecase/credit_upload.py sits alongside bureau normalization by building OCR-friendly tradeline aggregates before scoring.

Testing Considerations

What backend/tests/ml/test_variable_derivation.py covers

Test class Coverage
TestVariableMapStructure Validates tuple shape, source types, required-column rules, and set disjointness.
TestValidateFeatureCoverage Validates required-column detection, None handling, zero-value handling, sorted output, and optional-column coverage.
TestBalanceMismatches Validates balance-related derivation checks, coverage-score issues, and bureau-flag consistency.

High-value scenarios asserted by the tests

  • missing required columns are surfaced,
  • None values are treated as missing,
  • 0 remains valid,
  • bureau lineage is optional and must not leak into the required set,
  • salary, affordability, coverage, balance, and bureau mismatches are detected as human-readable issues.

Dependencies

  • pandas and numpy for feature assembly and numeric normalization.
  • dataclasses, datetime, typing, and pathlib for canonical data containers and utility flow.
  • joblib, lightgbm, xgboost, and scikit-learn in the model-training path inside credit_model.py.
  • pytest in backend/tests/ml/test_variable_derivation.py.
  • Vertex AI and runtime scoring infrastructure consume the calibrated score contract but are downstream of the feature-lineage layer.

Key Classes Reference

Class Responsibility
credit_canonical_schema.py Defines canonical bureau, open-banking, and application-level input containers.
credit_adapters.py Normalizes raw provider payloads into canonical dataclasses.
credit_variable_map.py Declares source-to-feature lineage and validation helpers.
open_banking_features.py Builds open-banking feature rows from transaction and account data.
credit_model.py Defines bureau-side feature contracts and derived feature engineering.
lita_global_score_converter.py Calibrates probability outputs into legacy and Lita global score scales.
credit_pipeline_validation.py Applies completeness scoring and gating rules before scoring.
schemas.py Defines the calibrated score response contract.
test_variable_derivation.py Verifies lineage, coverage, and derivation mismatch behavior.