Credit Scoring and Pipeline Domain
backend/src/ml/credit_canonical_schema.py, backend/src/ml/credit_adapters.py, backend/src/ml/credit_variable_map.py, backend/src/ml/credit_pipeline_validation.py, backend/tests/ml/test_adapters.py, backend/tests/ml/test_variable_derivation.py
Overview
This domain defines the canonical credit input used by the scoring pipeline before any model-specific feature engineering runs. Bureau payloads and open-banking payloads are first normalized into provider-agnostic dataclasses, so downstream scoring code can consume a stable shape instead of raw TransUnion, Equifax, Plaid, Salt Edge, or mixed provider responses.
The main business value is lineage control. VARIABLE_MAP ties each canonical field to one or more model feature columns, detect_derivation_mismatches() checks that feature vectors are logically consistent, and DataCompletenessReport captures whether an application is eligible to proceed to scoring. Together, these modules make it possible to audit a score from raw provider data back to the canonical source fields that produced it.
Architecture Overview
flowchart TB
subgraph ProviderPayloads [Provider Payloads]
TU[TransUnion row]
EQ[Equifax payload]
MR[Manual bureau report]
OB[Connected accounts payload]
end
subgraph CanonicalLayer [Canonical Credit Layer]
BA[BureauAdapter]
OA[OpenBankingAdapter]
CT[CanonicalTransaction]
CB[CanonicalBureauData]
CO[CanonicalOpenBankingData]
CI[CanonicalCreditInput]
end
subgraph LineageControls [Lineage and Validation Controls]
VM[VARIABLE_MAP]
VFC[validate_feature_coverage]
DDM[detect_derivation_mismatches]
CC[compute_completeness]
GT[gate]
DCR[DataCompletenessReport]
end
subgraph DownstreamScoring [Downstream Scoring]
FE[Feature engineering]
MODEL[Credit model]
end
TU --> BA
EQ --> BA
MR --> BA
OB --> OA
BA --> CB
OA --> CO
CO --> CT
CB --> CI
CO --> CI
CI --> CC
CC --> DCR
DCR --> GT
CI --> DDM
CI --> FE
CB --> FE
CO --> FE
VM --> FE
FE --> MODEL
VFC --> VM
Canonical Schema
Canonical Transaction
backend/src/ml/credit_canonical_schema.py
CanonicalTransaction is the atomic open-banking transaction record used after normalization. It stores currency-adjusted transaction data in a provider-neutral shape so later feature extraction can rely on a consistent schema.
| Property | Type | Description |
|---|---|---|
date |
`datetime \ | None` |
amount_usd |
float |
Amount converted to USD at adapter time. |
direction |
Literal["credit", "debit"] |
Normalized transaction direction. |
category |
str |
Lower-cased provider-normalized category or label. |
counterparty_hash |
str |
Hashed or anonymized counterparty identifier. |
Canonical Bureau Data
backend/src/ml/credit_canonical_schema.py
CanonicalBureauData is the provider-agnostic bureau summary used by the pipeline. It captures score, utilization, delinquency, age, inquiries, and risk flags from any supported bureau source, plus a raw payload for auditability.
| Property | Type | Description |
|---|---|---|
provider |
str |
Source label such as transunion, equifax, manual_upload, or global. |
score_value |
`float \ | None` |
score_scale_min |
float |
Lower bound of the provider score scale. |
score_scale_max |
float |
Upper bound of the provider score scale. |
score_provider |
str |
Provider score family such as vantage_4, fico_8, or equifax_risk. |
utilization_ratio |
`float \ | None` |
delinquencies_24m |
int |
Delinquency event count in the last 24 months. |
worst_dpd_24m |
int |
Worst days-past-due bucket in the last 24 months. |
credit_age_months |
`float \ | None` |
credit_age_months_oldest |
`float \ | None` |
inquiries_12m |
int |
Inquiry count in the last 12 months. |
new_accounts_12m |
int |
New account count in the last 12 months. |
collections_flag |
bool |
Collections presence flag. |
collections_count |
int |
Collections count. |
default_flag |
bool |
Default flag. |
report_recency_days |
float |
Days since the bureau report was pulled. |
parse_confidence |
float |
OCR or parser confidence in the 0–1 range. |
fraud_risk_flag |
bool |
Fraud risk indicator. |
aml_risk_flag |
bool |
AML risk indicator. |
estimated_income_score |
`float \ | None` |
revolving_util_patterns |
dict[str, float] |
TransUnion revolving utilization trend buckets. |
bankcard_util_patterns |
dict[str, float] |
TransUnion bankcard utilization trend buckets. |
trade_age_months |
dict[str, float] |
TransUnion trade-age buckets by trade class. |
loan_balances |
dict[str, float] |
TransUnion loan balance buckets. |
raw |
dict[str, Any] |
Original provider payload preserved for audit. |
Canonical Open Banking Data
backend/src/ml/credit_canonical_schema.py
CanonicalOpenBankingData aggregates connected-account data into a provider-agnostic banking view. It stores institutions, account counts, transaction history, and USD-normalized balances so the feature builder can operate without provider-specific branching.
| Property | Type | Description |
|---|---|---|
provider |
str |
Provider label such as plaid, saltedge, finverse, manual, mixed, or the adapter’s fallback value. |
institutions |
list[str] |
Linked provider or institution names. |
accounts_count |
int |
Total normalized account count. |
days_covered |
int |
Calendar span covered by the transaction history. |
transactions_90d |
list[CanonicalTransaction] |
Normalized transactions within the last 90 days. |
balance_snapshots_usd |
list[float] |
USD-normalized balance readings. |
raw |
dict[str, Any] |
Original provider payload preserved for audit. |
Data Completeness Report
[!NOTE]
OpenBankingAdapter.from_connected_accounts()can returnprovider="unknown"when the input is not a dict or list, even though the class comment highlights the usual labels. Downstream code should treat the provider field as a runtime value, not a closed enumeration.
backend/src/ml/credit_canonical_schema.py
DataCompletenessReport captures the validation result for a scoring request. It is produced by the pipeline validation layer and attached to CanonicalCreditInput so the scoring decision can be traced to concrete completeness conditions.
| Property | Type | Description |
|---|---|---|
bureau_available |
bool |
Whether bureau data is present. |
bureau_recency_ok |
bool |
Whether the bureau report age is within BUREAU_MAX_AGE_DAYS. |
banking_available |
bool |
Whether open-banking data is present. |
days_covered_ok |
bool |
Whether banking history meets MIN_DAYS_COVERED. |
salary_detected |
bool |
Whether salary activity was detected. |
income_verified |
bool |
Whether income is verified by salary, bureau income, or stated income. |
kyc_passed |
bool |
Whether KYC passed. |
flags |
list[str] |
Validation flags collected during completeness checks. |
completeness_score |
float |
Composite completeness score in the 0–1 range. |
gate_result |
Literal["pass", "refer", "decline"] |
Final gating decision. |
Canonical Credit Input
backend/src/ml/credit_canonical_schema.py
CanonicalCreditInput is the unified input object that holds application, bureau, banking, and macro context. It is the handoff point between provider normalization and feature generation, and it carries the completeness report after validation.
| Property | Type | Description |
|---|---|---|
application_id |
str |
Application identifier used throughout the scoring flow. |
bureau |
`CanonicalBureauData \ | None` |
banking |
`CanonicalOpenBankingData \ | None` |
income_stated_usd |
float |
Stated income in USD. |
savings_usd |
float |
Stated savings in USD. |
budget_usd |
float |
Stated budget in USD. |
employment_status |
str |
Employment status text used by feature engineering. |
home_country |
str |
Borrower home country. |
host_country |
str |
Scoring corridor host country. |
product_type |
str |
Product type, defaulting to rental. |
kyc_passed |
bool |
KYC outcome. |
host_unemployment_rate |
float |
Macro unemployment context. |
host_inflation_rate |
float |
Macro inflation context. |
host_interest_rate_proxy |
float |
Macro interest-rate proxy. |
data_completeness |
DataCompletenessReport |
Validation and gating outcome attached after completeness checks. |
Provider Adapters
Bureau Adapter
backend/src/ml/credit_adapters.py
BureauAdapter converts bureau-specific payloads into CanonicalBureauData. It supports the normalized TransUnion row path, Equifax instant-intake payloads, and manually uploaded or OCR-extracted bureau reports.
| Method | Description |
|---|---|
from_transunion_row |
Normalizes a TransUnion batch row and accepts both raw field codes and renamed columns. |
from_equifax_payload |
Normalizes an Equifax instant-intake API payload into canonical bureau data. |
from_manual_report |
Normalizes a manually uploaded or OCR-extracted bureau report. |
TransUnion normalization behavior
from_transunion_row() explicitly checks both the renamed model-facing columns and the original bureau codes. For example, score fields such as VantageScore and evtg04_finscore are both accepted, with renamed columns taking priority when both are present.
It also derives bureau metrics from TransUnion-specific fields:
- delinquency recency windows from
Months_Since_Last_30D_DelinquencyandMonths_Since_Last_60Dplus_Delinquency - revolving and bankcard utilization trend buckets from
ead52uz2_rev*andead52uz2_bkc* - trade age buckets from
eads142_* - loan balance buckets from the
Past_12M_*andeads142_*fields
Equifax normalization behavior
from_equifax_payload() reads credit_profile.risk_score, credit_profile.tradelines, inquiries_12m, utilization_ratio, collections_count, default_flag, and top-level recency / confidence / risk flags. It counts delinquent tradelines where days_past_due > 0 and caps the worst DPD bucket at 180.
Manual report normalization behavior
from_manual_report() mirrors the keys used by the runtime uploader path. It accepts score_value or score, carries score_provider forward, and preserves the original report in raw.
Open Banking Adapter
backend/src/ml/credit_adapters.py
OpenBankingAdapter normalizes connected-account payloads into CanonicalOpenBankingData. It handles provider envelopes, currency conversion, 90-day transaction filtering, and provider inference from the incoming payload keys.
| Method | Description |
|---|---|
from_connected_accounts |
Normalizes the generic connected-accounts payload and infers the provider label. |
from_saltedge |
Normalizes a Salt Edge data envelope by reusing the connected-accounts path. |
Provider-normalization strategy
from_connected_accounts() treats the top-level input as either a dict, a list, or a single provider object wrapped into a dict. It skips bureau-side keys in _BUREAU_KEYS so bureau data embedded in the same envelope is not double-counted as banking data.
Provider inference is driven by the seen provider names:
plaidproducesprovider="plaid"- Salt Edge variants produce
provider="saltedge" finverseproducesprovider="finverse"- one provider name is reused directly
- multiple provider names collapse to
provider="mixed"
Banking data mapping
The adapter maps each account into:
- account count
- institution names
- balance snapshots in USD
- normalized transactions in the last 90 days
Transactions are filtered to the 90-day window, converted to absolute USD amounts, lower-cased by category, and normalized to credit or debit direction. Balance values are converted through _DEFAULT_FX_TO_USD.
Salt Edge envelope handling
from_saltedge() reads payload["data"] when present, extracts connection.provider_name or connection.country_code as the provider hint, and delegates to from_connected_accounts(). It then forces the resulting provider label to saltedge.
Adapter Flow
sequenceDiagram
participant P as Provider Payload
participant BA as BureauAdapter
participant OA as OpenBankingAdapter
participant CB as CanonicalBureauData
participant CO as CanonicalOpenBankingData
participant CI as CanonicalCreditInput
P->>BA: TransUnion row
BA->>CB: from_transunion_row
P->>BA: Equifax payload
BA->>CB: from_equifax_payload
P->>BA: Manual report
BA->>CB: from_manual_report
P->>OA: Connected accounts
OA->>CO: from_connected_accounts
P->>OA: Salt Edge envelope
OA->>CO: from_saltedge
CB->>CI: attach bureau
CO->>CI: attach banking
Feature Lineage Controls
Variable Map
backend/src/ml/credit_variable_map.py
VARIABLE_MAP is the authoritative lineage table for the credit-scoring pipeline. Each tuple records which canonical path produces which model feature column, what source family it belongs to, and whether the feature is required for a valid score.
| Tuple Element | Meaning |
|---|---|
canonical_path |
Dotted path into CanonicalCreditInput, such as bureau.score_value or banking.transactions_90d. |
feature_column |
Output feature key that enters the model feature dict. |
source_type |
One of bureau, banking, application, or macro. |
required |
True means the feature must exist for a valid score; False means it is optional. |
The map covers four source families:
- application and KYC fields
- macro and corridor context
- bureau features, including score, risk, quality, and TransUnion trended fields
- open-banking features, including account metadata, income, spending, affordability, balances, and liquidity
Derived Lineage Sets
backend/src/ml/credit_variable_map.py
The module precomputes lineage indexes from VARIABLE_MAP:
REQUIRED_FEATURE_COLUMNSBANKING_DERIVED_FEATURESBUREAU_DERIVED_FEATURESALL_MAPPED_FEATURESCANONICAL_SOURCES
These sets are used to validate coverage, isolate bureau-derived columns, and inventory which canonical paths feed the model.
Lineage Validation
detect_derivation_mismatches() inspects a built feature dictionary for internal inconsistencies. It is designed to catch cases where canonical data was present but the corresponding feature translation failed or leaked stale values from a previous run.
| Function | Description |
|---|---|
detect_derivation_mismatches |
Flags logical inconsistencies in the feature dictionary. |
validate_feature_coverage |
Validates that model-facing feature columns are covered by the lineage map. |
list_unmapped_model_columns |
Identifies model columns that are not represented in the lineage map. |
list_mapped_but_unused_columns |
Identifies map entries that are not used by the model. |
Mismatch checks implemented in detect_derivation_mismatches()
The function explicitly checks:
- salary consistency
salary_detected=Truebutsalary_payer_count_90d=0salary_detected=Truebutsalary_payment_count_90d=0salary_detected=Truebutsalary_regularity_score=0.0salary_detected=Falsebutsalary_payer_count_90d>0
- non-salary income consistency
non_salary_income_share=0.0when banking data is present and no salary is detectedincome_sources_count>1when salary is not detectedincome_value=0.0despite banking data being present
- bureau consistency
- bureau data exists but
bureau_available_flag=0 bureau_available_flag=1butbureau_score_value=0- bureau score present without bureau data in canonical input
- bureau data exists but
- affordability consistency
affordability_rationot aligned withbudget_value / income_valueincome_value>0butbudget_value=0
- coverage and balance consistency
- banking present but
coverage_score=0.0 days_covered>0butcoverage_score=0.0- banking present but
avg_balance_90d=0.0 min_balance_30d > avg_balance_30d
- banking present but
Lineage Control Flow
flowchart TD
CI[CanonicalCreditInput] --> VM[VARIABLE_MAP]
VM --> FC[Feature Columns]
CI --> FE[Feature Builder]
FE --> FD[Feature Dict]
FD --> DM[detect_derivation_mismatches]
FD --> VC[validate_feature_coverage]
DM --> PASS[Lineage Check Result]
VC --> PASS
Validation and Gating
Completeness and Gating Rules
backend/src/ml/credit_pipeline_validation.py
This module decides whether an application should be scored before any model call is made. It produces a DataCompletenessReport, applies deterministic gating rules, and attaches the result to CanonicalCreditInput.
Thresholds and rule constants
| Constant | Description |
|---|---|
MIN_DAYS_COVERED |
Minimum banking history days required for coverage. |
BUREAU_MAX_AGE_DAYS |
Maximum bureau report age allowed before the report is considered stale. |
MIN_COMPLETENESS_SCORE_TO_PASS |
Minimum completeness score required to pass. |
_BUREAU_REQUIRED_PRODUCTS |
Product types that require bureau data, defaulting to high_ticket_loan. |
_DECLINE_FLAGS |
KYC_NOT_PASSED, FRAUD_RISK_DETECTED, AML_RISK_DETECTED. |
_REFER_FLAGS |
BUREAU_REQUIRED_BUT_ABSENT, NO_BANKING_DATA, INSUFFICIENT_HISTORY. |
Public functions
| Function | Description |
|---|---|
compute_completeness |
Builds a DataCompletenessReport from CanonicalCreditInput. |
gate |
Applies completeness gating and attaches the report to the canonical input. |
Gating priority
- Decline if
KYC_NOT_PASSED,FRAUD_RISK_DETECTED, orAML_RISK_DETECTED - Refer if
BUREAU_REQUIRED_BUT_ABSENT,NO_BANKING_DATA, orINSUFFICIENT_HISTORY - Pass in all other cases
Completeness score composition
compute_completeness() builds the composite score from these weighted conditions:
kyc_passed— 0.20banking_available— 0.20days_covered_ok— 0.15salary_detected— 0.10income_verified— 0.10bureau_available— 0.15bureau_recency_okwhen bureau data is available — 0.10
It also:
- marks income as verified when salary, bureau income, or stated income is present
- appends
BUREAU_REQUIRED_BUT_ABSENTwhen the product type requires bureau data - stores flags and final
gate_resulton the report
Validation Flow
sequenceDiagram
participant CI as CanonicalCreditInput
participant CV as compute_completeness
participant DR as DataCompletenessReport
participant GT as gate
CI->>CV: canonical input
CV->>DR: evaluate bureau, banking, KYC, income
CV->>DR: assign completeness_score
CV->>DR: assign flags and gate_result
GT->>CI: attach report to data_completeness
Test Coverage
Adapter Tests
backend/tests/ml/test_adapters.py
The adapter tests verify the normalization contract for both bureau and open-banking paths. They cover:
- TransUnion score mapping using renamed and raw field codes
- priority of renamed fields over raw codes
- estimated income mapping
- oldest trade age and delinquency derivations
- loan balance extraction
- raw payload preservation
- non-numeric field skipping
- Equifax score and tradeline mapping
- fraud and AML flag propagation
- manual report parsing
- provider inference and account-history handling for open-banking payloads
- currency conversion and history-window filtering
- bureau-key exclusion from banking ingestion
Variable Derivation Tests
backend/tests/ml/test_variable_derivation.py
The lineage tests verify that the feature map remains aligned with the model feature set. They exercise:
- structural invariants of
VARIABLE_MAP - required vs optional feature coverage
- detection of unmapped model columns
- detection of mapped but unused columns
- mismatch detection for salary, bureau, balance, and affordability derivations
Pipeline Validation Tests
backend/tests/ml/test_pipeline_validation.py
The validation tests verify:
- decline gates for KYC, fraud, and AML
- refer gates for missing banking data and insufficient history
- bureau-required product behavior
- completeness score weighting
- stale bureau handling
- salary detection from transaction categories
- income-verification combinations
- attachment of the completeness report to
CanonicalCreditInput
Error Handling
The adapters and validation helpers use defensive coercion and preserve audit context. The bureau adapters silently skip non-numeric values during field extraction, and the open-banking adapter returns an unknown provider label when the input cannot be normalized as a provider envelope.
The validation layer encodes errors as flags and gating outcomes instead of exceptions for expected data-quality problems. That allows the pipeline to distinguish between a hard decline, a referral, and a pass while still retaining the reason trail in DataCompletenessReport.flags.
Dependencies
dataclassesfor canonical schema recordsdatetimefor transaction timestamps and recency calculationstypingfor provider-neutral annotations andLiteralvaluesloggingincredit_adapters.pyfor adapter diagnosticssrc.ml.credit_canonical_schemafor shared canonical dataclassessrc.ml.credit_variable_mapfor lineage validation and feature mappingsrc.ml.credit_pipeline_validationfor completeness and gatingbackend/tests/ml/test_adapters.pyandbackend/tests/ml/test_variable_derivation.pyfor contract verification
Integration Points
credit_score_runtimeconsumesCanonicalCreditInput, banking aggregates, and bureau payloads after adapter normalization.- Feature engineering reads canonical fields through
VARIABLE_MAPso new providers only require new adapters. - Model-facing feature sets rely on the lineage controls to keep canonical paths, feature columns, and gating rules auditable.
- Tests in
backend/tests/ml/guard provider normalization, completeness logic, and feature-lineage integrity.
Key Classes Reference
| Class | Responsibility |
|---|---|
CanonicalTransaction |
Normalized USD transaction record for open-banking history. |
CanonicalBureauData |
Provider-agnostic bureau summary with preserved raw payload. |
CanonicalOpenBankingData |
Provider-agnostic banking summary with normalized transactions and balances. |
DataCompletenessReport |
Completeness and gating summary used before scoring. |
CanonicalCreditInput |
Unified credit input containing bureau, banking, application, and macro context. |
BureauAdapter |
Normalizes bureau provider payloads into canonical bureau data. |
OpenBankingAdapter |
Normalizes connected-account payloads into canonical open-banking data. |