Banking and Financial Data Domain
Overview
This domain turns provider-specific open-banking payloads into a canonical banking representation that can be consumed by both credit scoring and dashboard analytics. The backend normalizes Plaid, Salt Edge, and generic connected-account structures into CanonicalOpenBankingData, then derives stable feature vectors for the scoring pipeline. On the frontend, financial insight helpers reshape the same connected-account data into dashboard-friendly records without depending on provider-specific field names.
The business value is consistency: the scoring stack can count linked institutions, linked accounts, days covered, transaction activity, and USD-normalized balances from one shape, while the UI can aggregate the same data for transaction timelines and financial summaries. The canonical layer also supports completeness gating through DataCompletenessReport, which records whether banking history is sufficient for downstream scoring decisions.
Architecture Overview
flowchart TB
subgraph BackendScoring[Backend scoring]
PlaidPayload[Plaid payload]
SaltEdgePayload[Salt Edge payload]
ConnectedAccounts[Connected accounts payload]
OpenBankingAdapter[OpenBankingAdapter]
CanonicalData[CanonicalOpenBankingData]
Completeness[DataCompletenessReport]
FeatureVector[build_open_banking_feature_vector]
BalanceFeatures[build_balance_features]
ModelAlignment[credit_model open banking alignment]
PlaidPayload --> OpenBankingAdapter
SaltEdgePayload --> OpenBankingAdapter
ConnectedAccounts --> OpenBankingAdapter
OpenBankingAdapter --> CanonicalData
CanonicalData --> Completeness
CanonicalData --> FeatureVector
BalanceFeatures --> FeatureVector
FeatureVector --> ModelAlignment
end
subgraph FrontendInsights[Frontend insights]
BackendApplication[BackendApplication connected_accounts]
NormalizeAccounts[normalizeConnectedAccounts]
NormalizeProvider[normalizeProvider]
DescriptorNormalize[normaliseDescriptorValue]
InsightHelpers[Financial insights and risk helpers]
BackendApplication --> NormalizeAccounts
NormalizeAccounts --> InsightHelpers
NormalizeProvider --> InsightHelpers
DescriptorNormalize --> InsightHelpers
end
Component Structure
1. Canonical Schema and Completeness Reporting
CanonicalOpenBankingData
backend/src/ml/credit_canonical_schema.py
Provider-agnostic banking container used by the scoring pipeline and analytics layers. The schema keeps provider identity, institution coverage, account counts, a bounded transaction-history window, and USD-normalized balance snapshots together in one object.
| Property | Type | Description |
|---|---|---|
provider |
str |
Provider label. The documented canonical values include plaid, saltedge, finverse, manual, mixed, and unknown for malformed input. |
institutions |
list[str] |
Normalized institution or provider labels collected from the payload. |
accounts_count |
int |
Number of normalized account records encountered across providers. |
days_covered |
int |
Calendar days spanned by the transaction history used for scoring. |
transactions_90d |
list[CanonicalTransaction] |
Transaction records retained for the 90-day scoring window. |
balance_snapshots_usd |
list[float] |
USD-converted balance readings collected from account payloads. |
raw |
dict[str, Any] |
Original provider payload preserved for audit and traceability. |
Coverage in this schema is centered on three fields: institutions, accounts_count, and days_covered. Those values feed both scoring completeness checks and frontend aggregation logic.
CanonicalBureauData
backend/src/ml/credit_canonical_schema.py
Supporting canonical bureau container used by CanonicalCreditInput. It provides the bureau-side fields that completeness gating and combined scoring can read alongside banking data.
| Property | Type | Description |
|---|---|---|
provider |
str |
Bureau source label such as transunion, equifax, manual_upload, or global. |
score_value |
`float \ | None` |
score_scale_min |
float |
Lower bound of the score scale. |
score_scale_max |
float |
Upper bound of the score scale. |
score_provider |
str |
Bureau score family label. |
utilization_ratio |
`float \ | None` |
delinquencies_24m |
int |
Count of delinquency events in the last 24 months. |
worst_dpd_24m |
int |
Worst days-past-due bucket in the last 24 months. |
credit_age_months |
`float \ | None` |
credit_age_months_oldest |
`float \ | None` |
inquiries_12m |
int |
Number of inquiries in the last 12 months. |
new_accounts_12m |
int |
Number of new accounts in the last 12 months. |
collections_flag |
bool |
Collection presence indicator. |
collections_count |
int |
Collection count. |
default_flag |
bool |
Default indicator. |
report_recency_days |
float |
Days since the bureau report was pulled. |
parse_confidence |
float |
Parser or OCR confidence score. |
fraud_risk_flag |
bool |
Fraud risk indicator. |
aml_risk_flag |
bool |
AML risk indicator. |
estimated_income_score |
`float \ | None` |
revolving_util_patterns |
dict[str, float] |
Trended revolving-utilization pattern counts. |
bankcard_util_patterns |
dict[str, float] |
Trended bankcard-utilization pattern counts. |
trade_age_months |
dict[str, float] |
Trade-age buckets keyed by trade family. |
loan_balances |
dict[str, float] |
Loan balance buckets keyed by loan family. |
raw |
dict[str, Any] |
Original bureau payload preserved for audit. |
DataCompletenessReport
backend/src/ml/credit_canonical_schema.py
Structured completeness state used before scoring proceeds. It records which major data sources are present, whether coverage thresholds are met, and which gating flags were raised.
| Property | Type | Description |
|---|---|---|
bureau_available |
bool |
Whether bureau data is present. |
bureau_recency_ok |
bool |
Whether bureau data age is within the configured limit. |
banking_available |
bool |
Whether banking data is present. |
days_covered_ok |
bool |
Whether transaction coverage meets the configured minimum. |
salary_detected |
bool |
Whether salary activity was detected in banking transactions. |
income_verified |
bool |
Whether income is verified by salary, bureau, or stated income. |
kyc_passed |
bool |
Whether KYC passed. |
flags |
list[str] |
Deterministic completeness and gating flags. |
completeness_score |
float |
Composite completeness score from 0 to 1. |
gate_result |
Literal["pass", "refer", "decline"] |
Final gate decision. |
The gate result is driven by the computed flags and completeness score. The documented states are pass, refer, and decline.
CanonicalCreditInput
backend/src/ml/credit_canonical_schema.py
Unified scoring input that combines bureau data, banking data, application declarations, and macro context.
| Property | Type | Description |
|---|---|---|
application_id |
str |
Application identifier used to tie all inputs together. |
bureau |
`CanonicalBureauData \ | None` |
banking |
`CanonicalOpenBankingData \ | None` |
income_stated_usd |
float |
Stated income in USD. |
savings_usd |
float |
Stated savings in USD. |
budget_usd |
float |
Stated budget in USD. |
employment_status |
str |
Applicant employment state. |
home_country |
str |
Applicant home country. |
host_country |
str |
Destination or host country. |
product_type |
str |
Product family, documented as rental, loan, or high_ticket_loan. |
kyc_passed |
bool |
KYC result passed into the canonical object. |
host_unemployment_rate |
float |
Host macro unemployment input. |
host_inflation_rate |
float |
Host macro inflation input. |
host_interest_rate_proxy |
float |
Host rate proxy input. |
data_completeness |
DataCompletenessReport |
Populated after validation. |
2. Open Banking Adapter
OpenBankingAdapter
backend/src/ml/credit_adapters.py
Static adapter that collapses Plaid, Salt Edge, and generic connected-account payloads into CanonicalOpenBankingData.
| Method | Description |
|---|---|
from_connected_accounts |
Normalizes a generic connected-accounts payload and infers provider, account count, transactions, balances, and days covered. |
from_plaid |
Merges Plaid account and transaction envelopes by account_id, then delegates to from_connected_accounts. |
from_saltedge |
Normalizes a Salt Edge data envelope by mapping its connection and accounts payload into the generic adapter path. |
Coverage normalization rules in from_connected_accounts
- Treats list input as
{"default": ...}. - Skips any top-level keys classified as bureau data via
_BUREAU_KEYS. - Appends every provider key to
institutions, then deduplicates them while preserving first-seen order. - Counts every dict-shaped account as one linked account.
- Captures balances from:
balanceslists withbalancebalances.currentbalance
- Converts every numeric balance to USD via
_DEFAULT_FX_TO_USD. - Keeps only transactions from the last 90 days.
- Infers provider label as:
plaidif any provider key matches Plaidsaltedgeif any provider key matches Salt Edge aliasesfinverseif the provider key isfinverse- the single provider key if only one provider exists
mixedotherwise
Provider-specific behavior
from_plaidmerges the top-levelaccountsandtransactionsarrays by account identifier before canonicalization.from_saltedgereadsdata.connection.provider_nameordata.connection.country_codeto label the source, then forwardsdata.accountsinto the generic adapter.result.provideris explicitly overwritten toplaidorsaltedgeafter the generic normalization step.
3. Open Banking Feature Engineering
build_balance_features
backend/src/ml/open_banking_features.py
Builds balance and cash-flow aggregates from an accounts DataFrame.
| Method | Description |
|---|---|
build_balance_features |
Computes balance, overdraft, bounce, and cash-flow aggregates from account history. |
Returned feature keys:
avg_daily_balancemin_balance_30doverdraft_countbounce_countcash_buffer_rationet_cashflow_monthsurplus_ratio
Behavior visible in the code:
- Reads
historical_balancesfrom the accounts frame. - Explodes nested list or array balances before converting them to floats.
- Derives
min_balance_30dfrom the last 30 balance observations. - Computes
cash_buffer_ratiofrom average balance and outflow. - Computes
surplus_ratiofrom net cash flow divided by inflow.
build_open_banking_feature_vector
backend/src/ml/open_banking_features.py
Composes the final one-row feature frame used by the scoring model.
| Method | Description |
|---|---|
build_open_banking_feature_vector |
Combines income, balance, spending, stability, and country features into a single DataFrame row. |
Inputs:
tx: transaction DataFrameaccounts: account DataFramecountry_code: country code stringextra_features: optional DataFrame concatenated onto the final row
Output shape:
- A one-row
pandas.DataFramebuilt from a merged feature dictionary.
build_behavior_risk
backend/src/ml/build_open_banking_csvs.py
| Method | Description |
|---|---|
build_behavior_risk |
Delegates transaction input to obf.build_spending_features(tx_df). |
This helper is a thin CSV-generation wrapper around the same open-banking feature stack used by scoring.
4. Frontend Financial Insight Helpers
normalizeConnectedAccounts
frontend/lib/financial-insights.ts
Normalizes the backend connected_accounts shape into dashboard-friendly account groups.
| Method | Description |
|---|---|
normalizeConnectedAccounts |
Converts the backend connected-account payload into a normalized category-to-accounts map. |
Visible behavior:
- Returns
{}for non-object input. - Iterates over top-level category entries.
- Skips categories whose values are not arrays.
- Skips malformed account entries that are not objects.
- Requires one of
id,account_id, orconnection_idto keep a record. - Derives display names from:
nameconnection_nameinstitution_name
- Derives account names from:
account_nameofficial_name
- Derives institution names from:
institution_nameconnection_name
- Reads balances from the visible balance fields in the helper body.
normaliseDescriptorValue
frontend/lib/financial-insights.ts
| Method | Description |
|---|---|
normaliseDescriptorValue |
Lowercases and strips accents, digits, underscores, punctuation, and redundant whitespace from a descriptor string. |
The helper preserves a lowercase fallback when sanitization removes every visible alphanumeric character.
5. Provider Alias Normalization
normalizeProvider
frontend/lib/credit-risk.ts
| Method | Description |
|---|---|
normalizeProvider |
Maps a provider string to a canonical open-banking provider identifier. |
Supported aliases in OPEN_BANKING_PROVIDER_ALIASES:
saltedgesalt_edgesalt edgeplaid
If the input is empty or does not map to one of those aliases, the helper returns null.
6. Downstream Feature Alignment
Open banking feature columns
backend/src/ml/credit_model.py
The open-banking feature inventory is shared by the runtime scorer and the unified model through OPEN_BANKING_FEATURE_COLUMNS and ALL_FEATURE_COLUMNS. The visible coverage-related columns include:
institutions_linked_countaccounts_linked_countdays_coveredlargest_gap_daystx_count_90dactive_days_ratiomulti_institution_flagbalance_days_ratiocoverage_scorecoverage_flags_count
These columns are the model-facing expansion of the canonical banking schema and are expected to stay aligned with the runtime feature builder.
Feature Flows
1. Provider Payload to Canonical Banking Data
sequenceDiagram
participant Provider as Plaid or Salt Edge
participant Adapter as OpenBankingAdapter
participant Canonical as CanonicalOpenBankingData
participant Features as build_open_banking_feature_vector
participant Model as credit_model
Provider->>Adapter: provider payload
Adapter->>Adapter: from_plaid or from_saltedge
Adapter->>Adapter: from_connected_accounts
Adapter->>Canonical: provider, institutions, accounts_count, days_covered, transactions_90d, balance_snapshots_usd
Canonical-->>Adapter: normalized object
Adapter->>Features: tx and accounts frames
Features->>Features: build_balance_features and other feature builders
Features-->>Model: one row feature DataFrame
Flow details:
- Plaid and Salt Edge payloads are first reshaped into a generic connected-account envelope.
OpenBankingAdapter.from_connected_accountsproduces provider-agnostic banking data.days_coveredis computed from the transaction date span and clamped to the 90-day scoring window.- Balance readings are normalized to USD before feature generation.
- The resulting feature frame enters the model input alignment in
credit_model.py.
2. Backend Connected Accounts to Frontend Insight Aggregation
sequenceDiagram
participant Backend as BackendApplication
participant Normalize as normalizeConnectedAccounts
participant Descriptor as normaliseDescriptorValue
participant ProviderAlias as normalizeProvider
participant Insights as Financial insights helpers
Backend->>Normalize: connected_accounts payload
Normalize->>Normalize: clean ids, names, institutions, balances
Normalize-->>Insights: normalized category map
Descriptor->>Insights: sanitized descriptor text
ProviderAlias->>Insights: canonical provider label
Insights-->>Backend: dashboard ready aggregates
Flow details:
- The frontend consumes backend application records that already contain connected-account payloads.
normalizeConnectedAccountsremoves malformed entries and standardizes account naming.normaliseDescriptorValueprovides stable string normalization for matching and grouping text descriptors.normalizeProvidercollapses Salt Edge naming variants into a single provider label.- The UI layer can then aggregate balances and transaction groupings without branching on provider-specific field names.
State Management
DataCompletenessReport gating state
bureau_available: bureau data is present and can contribute to scoring.banking_available: banking data is present.days_covered_ok: transaction history meets the minimum configured coverage window.salary_detected: salary activity is visible in the banking stream.income_verified: income is verified by salary, bureau, or stated income.kyc_passed: KYC input is accepted.flags: deterministic gating markers such asINCOME_UNVERIFIED,BUREAU_REQUIRED_BUT_ABSENT,NO_BANKING_DATA, andINSUFFICIENT_HISTORY.completeness_score: weighted score used to decide whether the case can pass or should be referred.gate_result: final state, one ofpass,refer, ordecline.
The visible completeness logic is driven by presence and coverage signals rather than a mutable workflow object. Banking history becomes actionable only when the canonical object carries enough coverage to satisfy the gating rule set.
Integration Points
backend/src/ml/credit_adapters.pyconverts provider payloads into canonical banking data.backend/src/ml/credit_canonical_schema.pydefines the canonical objects consumed by scoring and completeness gating.backend/src/ml/open_banking_features.pybuilds feature vectors from normalized banking history.backend/src/ml/build_open_banking_csvs.pyreuses the same feature logic for CSV generation.backend/src/ml/credit_model.pymerges banking features into the unified model input set.backend/src/ml/credit_pipeline_validation.pyuses the canonical banking object to determine pass, refer, or decline outcomes.frontend/lib/financial-insights.tsnormalizes account payloads for dashboard aggregation.frontend/lib/credit-risk.tscanonicalizes provider names used by UI-side scoring helpers.
Error Handling
OpenBankingAdapter.from_connected_accountsreturnsCanonicalOpenBankingData(provider="unknown")when the input is not a dict or list.OpenBankingAdapter.from_connected_accountstolerates malformed provider payloads by skipping non-dict account entries and non-numeric amounts.OpenBankingAdapter.from_plaidandfrom_saltedgepreserve canonicalization even when the provider-specific top-level envelope differs.build_balance_featuresfalls back to zero-valued aggregates when balances are missing or empty.normalizeConnectedAccountsdrops malformed account entries instead of failing the full normalization pass.normalizeProviderreturnsnullfor empty or unmapped provider strings.DataCompletenessReportcarries the failure state forward throughflagsandgate_resultinstead of throwing during normalization.
Dependencies
- Backend feature engineering uses
pandasandnumpy. - Canonical schema objects use Python
dataclasses,datetime, andtyping. - Downstream model alignment is shared with
backend/src/ml/credit_model.pyandbackend/src/ml/credit_pipeline_validation.py. - Frontend normalization helpers use TypeScript string and collection primitives, plus Next.js application data shapes from the surrounding frontend codebase.
- Provider alias normalization in
frontend/lib/credit-risk.tsshares the same provider vocabulary as the backend adapter layer.
Testing Considerations
The visible tests exercise the following behaviors:
- Plaid provider detection in
OpenBankingAdapter.from_plaid. - Salt Edge provider detection in
OpenBankingAdapter.from_saltedge. - Account count aggregation across provider groups in
OpenBankingAdapter.from_connected_accounts. - Handling of
Nonepayloads and bare list payloads. - Category lowercasing for transaction records.
- Positive USD conversion for transaction amounts.
- GBP balance conversion using
_DEFAULT_FX_TO_USD. - Transaction merging by
account_idin Plaid payloads. - Preservation of institution names in the canonical institutions list.
days_coveredcoverage behavior in canonical banking data construction.
Key Classes Reference
| Class | Responsibility |
|---|---|
credit_canonical_schema.py |
Canonical banking, bureau, and completeness schemas for scoring inputs. |
credit_adapters.py |
Provider-specific payload normalization into canonical banking data. |
open_banking_features.py |
Banking feature extraction for score generation. |
financial-insights.ts |
Frontend normalization of connected accounts and descriptor text. |
credit-risk.ts |
Provider alias canonicalization for UI-side risk helpers. |
credit_model.py |
Downstream model alignment for open-banking feature columns. |
credit_pipeline_validation.py |
Completeness gating and pass or refer or decline decisions. |