Credit Scoring and Pipeline Domain

backend/src/ml/credit_canonical_schema.py, backend/src/ml/credit_adapters.py, backend/src/ml/credit_variable_map.py, backend/src/ml/credit_pipeline_validation.py, backend/tests/ml/test_adapters.py, backend/tests/ml/test_variable_derivation.py

Overview

This domain defines the canonical credit input used by the scoring pipeline before any model-specific feature engineering runs. Bureau payloads and open-banking payloads are first normalized into provider-agnostic dataclasses, so downstream scoring code can consume a stable shape instead of raw TransUnion, Equifax, Plaid, Salt Edge, or mixed provider responses.

The main business value is lineage control. VARIABLE_MAP ties each canonical field to one or more model feature columns, detect_derivation_mismatches() checks that feature vectors are logically consistent, and DataCompletenessReport captures whether an application is eligible to proceed to scoring. Together, these modules make it possible to audit a score from raw provider data back to the canonical source fields that produced it.

Architecture Overview

flowchart TB
    subgraph ProviderPayloads [Provider Payloads]
        TU[TransUnion row]
        EQ[Equifax payload]
        MR[Manual bureau report]
        OB[Connected accounts payload]
    end

    subgraph CanonicalLayer [Canonical Credit Layer]
        BA[BureauAdapter]
        OA[OpenBankingAdapter]
        CT[CanonicalTransaction]
        CB[CanonicalBureauData]
        CO[CanonicalOpenBankingData]
        CI[CanonicalCreditInput]
    end

    subgraph LineageControls [Lineage and Validation Controls]
        VM[VARIABLE_MAP]
        VFC[validate_feature_coverage]
        DDM[detect_derivation_mismatches]
        CC[compute_completeness]
        GT[gate]
        DCR[DataCompletenessReport]
    end

    subgraph DownstreamScoring [Downstream Scoring]
        FE[Feature engineering]
        MODEL[Credit model]
    end

    TU --> BA
    EQ --> BA
    MR --> BA
    OB --> OA

    BA --> CB
    OA --> CO
    CO --> CT
    CB --> CI
    CO --> CI

    CI --> CC
    CC --> DCR
    DCR --> GT
    CI --> DDM

    CI --> FE
    CB --> FE
    CO --> FE
    VM --> FE
    FE --> MODEL

    VFC --> VM

Canonical Schema

Canonical Transaction

backend/src/ml/credit_canonical_schema.py

CanonicalTransaction is the atomic open-banking transaction record used after normalization. It stores currency-adjusted transaction data in a provider-neutral shape so later feature extraction can rely on a consistent schema.

Property Type Description
date `datetime \ None`
amount_usd float Amount converted to USD at adapter time.
direction Literal["credit", "debit"] Normalized transaction direction.
category str Lower-cased provider-normalized category or label.
counterparty_hash str Hashed or anonymized counterparty identifier.

Canonical Bureau Data

backend/src/ml/credit_canonical_schema.py

CanonicalBureauData is the provider-agnostic bureau summary used by the pipeline. It captures score, utilization, delinquency, age, inquiries, and risk flags from any supported bureau source, plus a raw payload for auditability.

Property Type Description
provider str Source label such as transunion, equifax, manual_upload, or global.
score_value `float \ None`
score_scale_min float Lower bound of the provider score scale.
score_scale_max float Upper bound of the provider score scale.
score_provider str Provider score family such as vantage_4, fico_8, or equifax_risk.
utilization_ratio `float \ None`
delinquencies_24m int Delinquency event count in the last 24 months.
worst_dpd_24m int Worst days-past-due bucket in the last 24 months.
credit_age_months `float \ None`
credit_age_months_oldest `float \ None`
inquiries_12m int Inquiry count in the last 12 months.
new_accounts_12m int New account count in the last 12 months.
collections_flag bool Collections presence flag.
collections_count int Collections count.
default_flag bool Default flag.
report_recency_days float Days since the bureau report was pulled.
parse_confidence float OCR or parser confidence in the 0–1 range.
fraud_risk_flag bool Fraud risk indicator.
aml_risk_flag bool AML risk indicator.
estimated_income_score `float \ None`
revolving_util_patterns dict[str, float] TransUnion revolving utilization trend buckets.
bankcard_util_patterns dict[str, float] TransUnion bankcard utilization trend buckets.
trade_age_months dict[str, float] TransUnion trade-age buckets by trade class.
loan_balances dict[str, float] TransUnion loan balance buckets.
raw dict[str, Any] Original provider payload preserved for audit.

Canonical Open Banking Data

backend/src/ml/credit_canonical_schema.py

CanonicalOpenBankingData aggregates connected-account data into a provider-agnostic banking view. It stores institutions, account counts, transaction history, and USD-normalized balances so the feature builder can operate without provider-specific branching.

Property Type Description
provider str Provider label such as plaid, saltedge, finverse, manual, mixed, or the adapter’s fallback value.
institutions list[str] Linked provider or institution names.
accounts_count int Total normalized account count.
days_covered int Calendar span covered by the transaction history.
transactions_90d list[CanonicalTransaction] Normalized transactions within the last 90 days.
balance_snapshots_usd list[float] USD-normalized balance readings.
raw dict[str, Any] Original provider payload preserved for audit.

Data Completeness Report

[!NOTE] OpenBankingAdapter.from_connected_accounts() can return provider="unknown" when the input is not a dict or list, even though the class comment highlights the usual labels. Downstream code should treat the provider field as a runtime value, not a closed enumeration.

backend/src/ml/credit_canonical_schema.py

DataCompletenessReport captures the validation result for a scoring request. It is produced by the pipeline validation layer and attached to CanonicalCreditInput so the scoring decision can be traced to concrete completeness conditions.

Property Type Description
bureau_available bool Whether bureau data is present.
bureau_recency_ok bool Whether the bureau report age is within BUREAU_MAX_AGE_DAYS.
banking_available bool Whether open-banking data is present.
days_covered_ok bool Whether banking history meets MIN_DAYS_COVERED.
salary_detected bool Whether salary activity was detected.
income_verified bool Whether income is verified by salary, bureau income, or stated income.
kyc_passed bool Whether KYC passed.
flags list[str] Validation flags collected during completeness checks.
completeness_score float Composite completeness score in the 0–1 range.
gate_result Literal["pass", "refer", "decline"] Final gating decision.

Canonical Credit Input

backend/src/ml/credit_canonical_schema.py

CanonicalCreditInput is the unified input object that holds application, bureau, banking, and macro context. It is the handoff point between provider normalization and feature generation, and it carries the completeness report after validation.

Property Type Description
application_id str Application identifier used throughout the scoring flow.
bureau `CanonicalBureauData \ None`
banking `CanonicalOpenBankingData \ None`
income_stated_usd float Stated income in USD.
savings_usd float Stated savings in USD.
budget_usd float Stated budget in USD.
employment_status str Employment status text used by feature engineering.
home_country str Borrower home country.
host_country str Scoring corridor host country.
product_type str Product type, defaulting to rental.
kyc_passed bool KYC outcome.
host_unemployment_rate float Macro unemployment context.
host_inflation_rate float Macro inflation context.
host_interest_rate_proxy float Macro interest-rate proxy.
data_completeness DataCompletenessReport Validation and gating outcome attached after completeness checks.

Provider Adapters

Bureau Adapter

backend/src/ml/credit_adapters.py

BureauAdapter converts bureau-specific payloads into CanonicalBureauData. It supports the normalized TransUnion row path, Equifax instant-intake payloads, and manually uploaded or OCR-extracted bureau reports.

Method Description
from_transunion_row Normalizes a TransUnion batch row and accepts both raw field codes and renamed columns.
from_equifax_payload Normalizes an Equifax instant-intake API payload into canonical bureau data.
from_manual_report Normalizes a manually uploaded or OCR-extracted bureau report.

TransUnion normalization behavior

from_transunion_row() explicitly checks both the renamed model-facing columns and the original bureau codes. For example, score fields such as VantageScore and evtg04_finscore are both accepted, with renamed columns taking priority when both are present.

It also derives bureau metrics from TransUnion-specific fields:

  • delinquency recency windows from Months_Since_Last_30D_Delinquency and Months_Since_Last_60Dplus_Delinquency
  • revolving and bankcard utilization trend buckets from ead52uz2_rev* and ead52uz2_bkc*
  • trade age buckets from eads142_*
  • loan balance buckets from the Past_12M_* and eads142_* fields

Equifax normalization behavior

from_equifax_payload() reads credit_profile.risk_score, credit_profile.tradelines, inquiries_12m, utilization_ratio, collections_count, default_flag, and top-level recency / confidence / risk flags. It counts delinquent tradelines where days_past_due > 0 and caps the worst DPD bucket at 180.

Manual report normalization behavior

from_manual_report() mirrors the keys used by the runtime uploader path. It accepts score_value or score, carries score_provider forward, and preserves the original report in raw.

Open Banking Adapter

backend/src/ml/credit_adapters.py

OpenBankingAdapter normalizes connected-account payloads into CanonicalOpenBankingData. It handles provider envelopes, currency conversion, 90-day transaction filtering, and provider inference from the incoming payload keys.

Method Description
from_connected_accounts Normalizes the generic connected-accounts payload and infers the provider label.
from_saltedge Normalizes a Salt Edge data envelope by reusing the connected-accounts path.

Provider-normalization strategy

from_connected_accounts() treats the top-level input as either a dict, a list, or a single provider object wrapped into a dict. It skips bureau-side keys in _BUREAU_KEYS so bureau data embedded in the same envelope is not double-counted as banking data.

Provider inference is driven by the seen provider names:

  • plaid produces provider="plaid"
  • Salt Edge variants produce provider="saltedge"
  • finverse produces provider="finverse"
  • one provider name is reused directly
  • multiple provider names collapse to provider="mixed"

Banking data mapping

The adapter maps each account into:

  • account count
  • institution names
  • balance snapshots in USD
  • normalized transactions in the last 90 days

Transactions are filtered to the 90-day window, converted to absolute USD amounts, lower-cased by category, and normalized to credit or debit direction. Balance values are converted through _DEFAULT_FX_TO_USD.

Salt Edge envelope handling

from_saltedge() reads payload["data"] when present, extracts connection.provider_name or connection.country_code as the provider hint, and delegates to from_connected_accounts(). It then forces the resulting provider label to saltedge.

Adapter Flow

sequenceDiagram
    participant P as Provider Payload
    participant BA as BureauAdapter
    participant OA as OpenBankingAdapter
    participant CB as CanonicalBureauData
    participant CO as CanonicalOpenBankingData
    participant CI as CanonicalCreditInput

    P->>BA: TransUnion row
    BA->>CB: from_transunion_row
    P->>BA: Equifax payload
    BA->>CB: from_equifax_payload
    P->>BA: Manual report
    BA->>CB: from_manual_report

    P->>OA: Connected accounts
    OA->>CO: from_connected_accounts
    P->>OA: Salt Edge envelope
    OA->>CO: from_saltedge

    CB->>CI: attach bureau
    CO->>CI: attach banking

Feature Lineage Controls

Variable Map

backend/src/ml/credit_variable_map.py

VARIABLE_MAP is the authoritative lineage table for the credit-scoring pipeline. Each tuple records which canonical path produces which model feature column, what source family it belongs to, and whether the feature is required for a valid score.

Tuple Element Meaning
canonical_path Dotted path into CanonicalCreditInput, such as bureau.score_value or banking.transactions_90d.
feature_column Output feature key that enters the model feature dict.
source_type One of bureau, banking, application, or macro.
required True means the feature must exist for a valid score; False means it is optional.

The map covers four source families:

  • application and KYC fields
  • macro and corridor context
  • bureau features, including score, risk, quality, and TransUnion trended fields
  • open-banking features, including account metadata, income, spending, affordability, balances, and liquidity

Derived Lineage Sets

backend/src/ml/credit_variable_map.py

The module precomputes lineage indexes from VARIABLE_MAP:

  • REQUIRED_FEATURE_COLUMNS
  • BANKING_DERIVED_FEATURES
  • BUREAU_DERIVED_FEATURES
  • ALL_MAPPED_FEATURES
  • CANONICAL_SOURCES

These sets are used to validate coverage, isolate bureau-derived columns, and inventory which canonical paths feed the model.

Lineage Validation

detect_derivation_mismatches() inspects a built feature dictionary for internal inconsistencies. It is designed to catch cases where canonical data was present but the corresponding feature translation failed or leaked stale values from a previous run.

Function Description
detect_derivation_mismatches Flags logical inconsistencies in the feature dictionary.
validate_feature_coverage Validates that model-facing feature columns are covered by the lineage map.
list_unmapped_model_columns Identifies model columns that are not represented in the lineage map.
list_mapped_but_unused_columns Identifies map entries that are not used by the model.

Mismatch checks implemented in detect_derivation_mismatches()

The function explicitly checks:

  • salary consistency
    • salary_detected=True but salary_payer_count_90d=0
    • salary_detected=True but salary_payment_count_90d=0
    • salary_detected=True but salary_regularity_score=0.0
    • salary_detected=False but salary_payer_count_90d>0
  • non-salary income consistency
    • non_salary_income_share=0.0 when banking data is present and no salary is detected
    • income_sources_count>1 when salary is not detected
    • income_value=0.0 despite banking data being present
  • bureau consistency
    • bureau data exists but bureau_available_flag=0
    • bureau_available_flag=1 but bureau_score_value=0
    • bureau score present without bureau data in canonical input
  • affordability consistency
    • affordability_ratio not aligned with budget_value / income_value
    • income_value>0 but budget_value=0
  • coverage and balance consistency
    • banking present but coverage_score=0.0
    • days_covered>0 but coverage_score=0.0
    • banking present but avg_balance_90d=0.0
    • min_balance_30d > avg_balance_30d

Lineage Control Flow

flowchart TD
    CI[CanonicalCreditInput] --> VM[VARIABLE_MAP]
    VM --> FC[Feature Columns]
    CI --> FE[Feature Builder]
    FE --> FD[Feature Dict]
    FD --> DM[detect_derivation_mismatches]
    FD --> VC[validate_feature_coverage]
    DM --> PASS[Lineage Check Result]
    VC --> PASS

Validation and Gating

Completeness and Gating Rules

backend/src/ml/credit_pipeline_validation.py

This module decides whether an application should be scored before any model call is made. It produces a DataCompletenessReport, applies deterministic gating rules, and attaches the result to CanonicalCreditInput.

Thresholds and rule constants

Constant Description
MIN_DAYS_COVERED Minimum banking history days required for coverage.
BUREAU_MAX_AGE_DAYS Maximum bureau report age allowed before the report is considered stale.
MIN_COMPLETENESS_SCORE_TO_PASS Minimum completeness score required to pass.
_BUREAU_REQUIRED_PRODUCTS Product types that require bureau data, defaulting to high_ticket_loan.
_DECLINE_FLAGS KYC_NOT_PASSED, FRAUD_RISK_DETECTED, AML_RISK_DETECTED.
_REFER_FLAGS BUREAU_REQUIRED_BUT_ABSENT, NO_BANKING_DATA, INSUFFICIENT_HISTORY.

Public functions

Function Description
compute_completeness Builds a DataCompletenessReport from CanonicalCreditInput.
gate Applies completeness gating and attaches the report to the canonical input.

Gating priority

  1. Decline if KYC_NOT_PASSED, FRAUD_RISK_DETECTED, or AML_RISK_DETECTED
  2. Refer if BUREAU_REQUIRED_BUT_ABSENT, NO_BANKING_DATA, or INSUFFICIENT_HISTORY
  3. Pass in all other cases

Completeness score composition

compute_completeness() builds the composite score from these weighted conditions:

  • kyc_passed — 0.20
  • banking_available — 0.20
  • days_covered_ok — 0.15
  • salary_detected — 0.10
  • income_verified — 0.10
  • bureau_available — 0.15
  • bureau_recency_ok when bureau data is available — 0.10

It also:

  • marks income as verified when salary, bureau income, or stated income is present
  • appends BUREAU_REQUIRED_BUT_ABSENT when the product type requires bureau data
  • stores flags and final gate_result on the report

Validation Flow

sequenceDiagram
    participant CI as CanonicalCreditInput
    participant CV as compute_completeness
    participant DR as DataCompletenessReport
    participant GT as gate

    CI->>CV: canonical input
    CV->>DR: evaluate bureau, banking, KYC, income
    CV->>DR: assign completeness_score
    CV->>DR: assign flags and gate_result
    GT->>CI: attach report to data_completeness

Test Coverage

Adapter Tests

backend/tests/ml/test_adapters.py

The adapter tests verify the normalization contract for both bureau and open-banking paths. They cover:

  • TransUnion score mapping using renamed and raw field codes
  • priority of renamed fields over raw codes
  • estimated income mapping
  • oldest trade age and delinquency derivations
  • loan balance extraction
  • raw payload preservation
  • non-numeric field skipping
  • Equifax score and tradeline mapping
  • fraud and AML flag propagation
  • manual report parsing
  • provider inference and account-history handling for open-banking payloads
  • currency conversion and history-window filtering
  • bureau-key exclusion from banking ingestion

Variable Derivation Tests

backend/tests/ml/test_variable_derivation.py

The lineage tests verify that the feature map remains aligned with the model feature set. They exercise:

  • structural invariants of VARIABLE_MAP
  • required vs optional feature coverage
  • detection of unmapped model columns
  • detection of mapped but unused columns
  • mismatch detection for salary, bureau, balance, and affordability derivations

Pipeline Validation Tests

backend/tests/ml/test_pipeline_validation.py

The validation tests verify:

  • decline gates for KYC, fraud, and AML
  • refer gates for missing banking data and insufficient history
  • bureau-required product behavior
  • completeness score weighting
  • stale bureau handling
  • salary detection from transaction categories
  • income-verification combinations
  • attachment of the completeness report to CanonicalCreditInput

Error Handling

The adapters and validation helpers use defensive coercion and preserve audit context. The bureau adapters silently skip non-numeric values during field extraction, and the open-banking adapter returns an unknown provider label when the input cannot be normalized as a provider envelope.

The validation layer encodes errors as flags and gating outcomes instead of exceptions for expected data-quality problems. That allows the pipeline to distinguish between a hard decline, a referral, and a pass while still retaining the reason trail in DataCompletenessReport.flags.

Dependencies

  • dataclasses for canonical schema records
  • datetime for transaction timestamps and recency calculations
  • typing for provider-neutral annotations and Literal values
  • logging in credit_adapters.py for adapter diagnostics
  • src.ml.credit_canonical_schema for shared canonical dataclasses
  • src.ml.credit_variable_map for lineage validation and feature mapping
  • src.ml.credit_pipeline_validation for completeness and gating
  • backend/tests/ml/test_adapters.py and backend/tests/ml/test_variable_derivation.py for contract verification

Integration Points

  • credit_score_runtime consumes CanonicalCreditInput, banking aggregates, and bureau payloads after adapter normalization.
  • Feature engineering reads canonical fields through VARIABLE_MAP so new providers only require new adapters.
  • Model-facing feature sets rely on the lineage controls to keep canonical paths, feature columns, and gating rules auditable.
  • Tests in backend/tests/ml/ guard provider normalization, completeness logic, and feature-lineage integrity.

Key Classes Reference

Class Responsibility
CanonicalTransaction Normalized USD transaction record for open-banking history.
CanonicalBureauData Provider-agnostic bureau summary with preserved raw payload.
CanonicalOpenBankingData Provider-agnostic banking summary with normalized transactions and balances.
DataCompletenessReport Completeness and gating summary used before scoring.
CanonicalCreditInput Unified credit input containing bureau, banking, application, and macro context.
BureauAdapter Normalizes bureau provider payloads into canonical bureau data.
OpenBankingAdapter Normalizes connected-account payloads into canonical open-banking data.