Banking and Financial Data Domain

Overview

This domain turns provider-specific open-banking payloads into a canonical banking representation that can be consumed by both credit scoring and dashboard analytics. The backend normalizes Plaid, Salt Edge, and generic connected-account structures into CanonicalOpenBankingData, then derives stable feature vectors for the scoring pipeline. On the frontend, financial insight helpers reshape the same connected-account data into dashboard-friendly records without depending on provider-specific field names.

The business value is consistency: the scoring stack can count linked institutions, linked accounts, days covered, transaction activity, and USD-normalized balances from one shape, while the UI can aggregate the same data for transaction timelines and financial summaries. The canonical layer also supports completeness gating through DataCompletenessReport, which records whether banking history is sufficient for downstream scoring decisions.

Architecture Overview

flowchart TB
  subgraph BackendScoring[Backend scoring]
    PlaidPayload[Plaid payload]
    SaltEdgePayload[Salt Edge payload]
    ConnectedAccounts[Connected accounts payload]
    OpenBankingAdapter[OpenBankingAdapter]
    CanonicalData[CanonicalOpenBankingData]
    Completeness[DataCompletenessReport]
    FeatureVector[build_open_banking_feature_vector]
    BalanceFeatures[build_balance_features]
    ModelAlignment[credit_model open banking alignment]

    PlaidPayload --> OpenBankingAdapter
    SaltEdgePayload --> OpenBankingAdapter
    ConnectedAccounts --> OpenBankingAdapter
    OpenBankingAdapter --> CanonicalData
    CanonicalData --> Completeness
    CanonicalData --> FeatureVector
    BalanceFeatures --> FeatureVector
    FeatureVector --> ModelAlignment
  end

  subgraph FrontendInsights[Frontend insights]
    BackendApplication[BackendApplication connected_accounts]
    NormalizeAccounts[normalizeConnectedAccounts]
    NormalizeProvider[normalizeProvider]
    DescriptorNormalize[normaliseDescriptorValue]
    InsightHelpers[Financial insights and risk helpers]

    BackendApplication --> NormalizeAccounts
    NormalizeAccounts --> InsightHelpers
    NormalizeProvider --> InsightHelpers
    DescriptorNormalize --> InsightHelpers
  end

Component Structure

1. Canonical Schema and Completeness Reporting

CanonicalOpenBankingData

backend/src/ml/credit_canonical_schema.py

Provider-agnostic banking container used by the scoring pipeline and analytics layers. The schema keeps provider identity, institution coverage, account counts, a bounded transaction-history window, and USD-normalized balance snapshots together in one object.

Property Type Description
provider str Provider label. The documented canonical values include plaid, saltedge, finverse, manual, mixed, and unknown for malformed input.
institutions list[str] Normalized institution or provider labels collected from the payload.
accounts_count int Number of normalized account records encountered across providers.
days_covered int Calendar days spanned by the transaction history used for scoring.
transactions_90d list[CanonicalTransaction] Transaction records retained for the 90-day scoring window.
balance_snapshots_usd list[float] USD-converted balance readings collected from account payloads.
raw dict[str, Any] Original provider payload preserved for audit and traceability.

Coverage in this schema is centered on three fields: institutions, accounts_count, and days_covered. Those values feed both scoring completeness checks and frontend aggregation logic.

CanonicalBureauData

backend/src/ml/credit_canonical_schema.py

Supporting canonical bureau container used by CanonicalCreditInput. It provides the bureau-side fields that completeness gating and combined scoring can read alongside banking data.

Property Type Description
provider str Bureau source label such as transunion, equifax, manual_upload, or global.
score_value `float \ None`
score_scale_min float Lower bound of the score scale.
score_scale_max float Upper bound of the score scale.
score_provider str Bureau score family label.
utilization_ratio `float \ None`
delinquencies_24m int Count of delinquency events in the last 24 months.
worst_dpd_24m int Worst days-past-due bucket in the last 24 months.
credit_age_months `float \ None`
credit_age_months_oldest `float \ None`
inquiries_12m int Number of inquiries in the last 12 months.
new_accounts_12m int Number of new accounts in the last 12 months.
collections_flag bool Collection presence indicator.
collections_count int Collection count.
default_flag bool Default indicator.
report_recency_days float Days since the bureau report was pulled.
parse_confidence float Parser or OCR confidence score.
fraud_risk_flag bool Fraud risk indicator.
aml_risk_flag bool AML risk indicator.
estimated_income_score `float \ None`
revolving_util_patterns dict[str, float] Trended revolving-utilization pattern counts.
bankcard_util_patterns dict[str, float] Trended bankcard-utilization pattern counts.
trade_age_months dict[str, float] Trade-age buckets keyed by trade family.
loan_balances dict[str, float] Loan balance buckets keyed by loan family.
raw dict[str, Any] Original bureau payload preserved for audit.

DataCompletenessReport

backend/src/ml/credit_canonical_schema.py

Structured completeness state used before scoring proceeds. It records which major data sources are present, whether coverage thresholds are met, and which gating flags were raised.

Property Type Description
bureau_available bool Whether bureau data is present.
bureau_recency_ok bool Whether bureau data age is within the configured limit.
banking_available bool Whether banking data is present.
days_covered_ok bool Whether transaction coverage meets the configured minimum.
salary_detected bool Whether salary activity was detected in banking transactions.
income_verified bool Whether income is verified by salary, bureau, or stated income.
kyc_passed bool Whether KYC passed.
flags list[str] Deterministic completeness and gating flags.
completeness_score float Composite completeness score from 0 to 1.
gate_result Literal["pass", "refer", "decline"] Final gate decision.

The gate result is driven by the computed flags and completeness score. The documented states are pass, refer, and decline.

CanonicalCreditInput

backend/src/ml/credit_canonical_schema.py

Unified scoring input that combines bureau data, banking data, application declarations, and macro context.

Property Type Description
application_id str Application identifier used to tie all inputs together.
bureau `CanonicalBureauData \ None`
banking `CanonicalOpenBankingData \ None`
income_stated_usd float Stated income in USD.
savings_usd float Stated savings in USD.
budget_usd float Stated budget in USD.
employment_status str Applicant employment state.
home_country str Applicant home country.
host_country str Destination or host country.
product_type str Product family, documented as rental, loan, or high_ticket_loan.
kyc_passed bool KYC result passed into the canonical object.
host_unemployment_rate float Host macro unemployment input.
host_inflation_rate float Host macro inflation input.
host_interest_rate_proxy float Host rate proxy input.
data_completeness DataCompletenessReport Populated after validation.

2. Open Banking Adapter

OpenBankingAdapter

backend/src/ml/credit_adapters.py

Static adapter that collapses Plaid, Salt Edge, and generic connected-account payloads into CanonicalOpenBankingData.

Method Description
from_connected_accounts Normalizes a generic connected-accounts payload and infers provider, account count, transactions, balances, and days covered.
from_plaid Merges Plaid account and transaction envelopes by account_id, then delegates to from_connected_accounts.
from_saltedge Normalizes a Salt Edge data envelope by mapping its connection and accounts payload into the generic adapter path.
Coverage normalization rules in from_connected_accounts
  • Treats list input as {"default": ...}.
  • Skips any top-level keys classified as bureau data via _BUREAU_KEYS.
  • Appends every provider key to institutions, then deduplicates them while preserving first-seen order.
  • Counts every dict-shaped account as one linked account.
  • Captures balances from:
    • balances lists with balance
    • balances.current
    • balance
  • Converts every numeric balance to USD via _DEFAULT_FX_TO_USD.
  • Keeps only transactions from the last 90 days.
  • Infers provider label as:
    • plaid if any provider key matches Plaid
    • saltedge if any provider key matches Salt Edge aliases
    • finverse if the provider key is finverse
    • the single provider key if only one provider exists
    • mixed otherwise
Provider-specific behavior
  • from_plaid merges the top-level accounts and transactions arrays by account identifier before canonicalization.
  • from_saltedge reads data.connection.provider_name or data.connection.country_code to label the source, then forwards data.accounts into the generic adapter.
  • result.provider is explicitly overwritten to plaid or saltedge after the generic normalization step.

3. Open Banking Feature Engineering

build_balance_features

backend/src/ml/open_banking_features.py

Builds balance and cash-flow aggregates from an accounts DataFrame.

Method Description
build_balance_features Computes balance, overdraft, bounce, and cash-flow aggregates from account history.

Returned feature keys:

  • avg_daily_balance
  • min_balance_30d
  • overdraft_count
  • bounce_count
  • cash_buffer_ratio
  • net_cashflow_month
  • surplus_ratio

Behavior visible in the code:

  • Reads historical_balances from the accounts frame.
  • Explodes nested list or array balances before converting them to floats.
  • Derives min_balance_30d from the last 30 balance observations.
  • Computes cash_buffer_ratio from average balance and outflow.
  • Computes surplus_ratio from net cash flow divided by inflow.

build_open_banking_feature_vector

backend/src/ml/open_banking_features.py

Composes the final one-row feature frame used by the scoring model.

Method Description
build_open_banking_feature_vector Combines income, balance, spending, stability, and country features into a single DataFrame row.

Inputs:

  • tx: transaction DataFrame
  • accounts: account DataFrame
  • country_code: country code string
  • extra_features: optional DataFrame concatenated onto the final row

Output shape:

  • A one-row pandas.DataFrame built from a merged feature dictionary.

build_behavior_risk

backend/src/ml/build_open_banking_csvs.py

Method Description
build_behavior_risk Delegates transaction input to obf.build_spending_features(tx_df).

This helper is a thin CSV-generation wrapper around the same open-banking feature stack used by scoring.

4. Frontend Financial Insight Helpers

normalizeConnectedAccounts

frontend/lib/financial-insights.ts

Normalizes the backend connected_accounts shape into dashboard-friendly account groups.

Method Description
normalizeConnectedAccounts Converts the backend connected-account payload into a normalized category-to-accounts map.

Visible behavior:

  • Returns {} for non-object input.
  • Iterates over top-level category entries.
  • Skips categories whose values are not arrays.
  • Skips malformed account entries that are not objects.
  • Requires one of id, account_id, or connection_id to keep a record.
  • Derives display names from:
    • name
    • connection_name
    • institution_name
  • Derives account names from:
    • account_name
    • official_name
  • Derives institution names from:
    • institution_name
    • connection_name
  • Reads balances from the visible balance fields in the helper body.

normaliseDescriptorValue

frontend/lib/financial-insights.ts

Method Description
normaliseDescriptorValue Lowercases and strips accents, digits, underscores, punctuation, and redundant whitespace from a descriptor string.

The helper preserves a lowercase fallback when sanitization removes every visible alphanumeric character.

5. Provider Alias Normalization

normalizeProvider

frontend/lib/credit-risk.ts

Method Description
normalizeProvider Maps a provider string to a canonical open-banking provider identifier.

Supported aliases in OPEN_BANKING_PROVIDER_ALIASES:

  • saltedge
  • salt_edge
  • salt edge
  • plaid

If the input is empty or does not map to one of those aliases, the helper returns null.

6. Downstream Feature Alignment

Open banking feature columns

backend/src/ml/credit_model.py

The open-banking feature inventory is shared by the runtime scorer and the unified model through OPEN_BANKING_FEATURE_COLUMNS and ALL_FEATURE_COLUMNS. The visible coverage-related columns include:

  • institutions_linked_count
  • accounts_linked_count
  • days_covered
  • largest_gap_days
  • tx_count_90d
  • active_days_ratio
  • multi_institution_flag
  • balance_days_ratio
  • coverage_score
  • coverage_flags_count

These columns are the model-facing expansion of the canonical banking schema and are expected to stay aligned with the runtime feature builder.

Feature Flows

1. Provider Payload to Canonical Banking Data

sequenceDiagram
  participant Provider as Plaid or Salt Edge
  participant Adapter as OpenBankingAdapter
  participant Canonical as CanonicalOpenBankingData
  participant Features as build_open_banking_feature_vector
  participant Model as credit_model

  Provider->>Adapter: provider payload
  Adapter->>Adapter: from_plaid or from_saltedge
  Adapter->>Adapter: from_connected_accounts
  Adapter->>Canonical: provider, institutions, accounts_count, days_covered, transactions_90d, balance_snapshots_usd
  Canonical-->>Adapter: normalized object
  Adapter->>Features: tx and accounts frames
  Features->>Features: build_balance_features and other feature builders
  Features-->>Model: one row feature DataFrame

Flow details:

  1. Plaid and Salt Edge payloads are first reshaped into a generic connected-account envelope.
  2. OpenBankingAdapter.from_connected_accounts produces provider-agnostic banking data.
  3. days_covered is computed from the transaction date span and clamped to the 90-day scoring window.
  4. Balance readings are normalized to USD before feature generation.
  5. The resulting feature frame enters the model input alignment in credit_model.py.

2. Backend Connected Accounts to Frontend Insight Aggregation

sequenceDiagram
  participant Backend as BackendApplication
  participant Normalize as normalizeConnectedAccounts
  participant Descriptor as normaliseDescriptorValue
  participant ProviderAlias as normalizeProvider
  participant Insights as Financial insights helpers

  Backend->>Normalize: connected_accounts payload
  Normalize->>Normalize: clean ids, names, institutions, balances
  Normalize-->>Insights: normalized category map
  Descriptor->>Insights: sanitized descriptor text
  ProviderAlias->>Insights: canonical provider label
  Insights-->>Backend: dashboard ready aggregates

Flow details:

  1. The frontend consumes backend application records that already contain connected-account payloads.
  2. normalizeConnectedAccounts removes malformed entries and standardizes account naming.
  3. normaliseDescriptorValue provides stable string normalization for matching and grouping text descriptors.
  4. normalizeProvider collapses Salt Edge naming variants into a single provider label.
  5. The UI layer can then aggregate balances and transaction groupings without branching on provider-specific field names.

State Management

DataCompletenessReport gating state

  • bureau_available: bureau data is present and can contribute to scoring.
  • banking_available: banking data is present.
  • days_covered_ok: transaction history meets the minimum configured coverage window.
  • salary_detected: salary activity is visible in the banking stream.
  • income_verified: income is verified by salary, bureau, or stated income.
  • kyc_passed: KYC input is accepted.
  • flags: deterministic gating markers such as INCOME_UNVERIFIED, BUREAU_REQUIRED_BUT_ABSENT, NO_BANKING_DATA, and INSUFFICIENT_HISTORY.
  • completeness_score: weighted score used to decide whether the case can pass or should be referred.
  • gate_result: final state, one of pass, refer, or decline.

The visible completeness logic is driven by presence and coverage signals rather than a mutable workflow object. Banking history becomes actionable only when the canonical object carries enough coverage to satisfy the gating rule set.

Integration Points

  • backend/src/ml/credit_adapters.py converts provider payloads into canonical banking data.
  • backend/src/ml/credit_canonical_schema.py defines the canonical objects consumed by scoring and completeness gating.
  • backend/src/ml/open_banking_features.py builds feature vectors from normalized banking history.
  • backend/src/ml/build_open_banking_csvs.py reuses the same feature logic for CSV generation.
  • backend/src/ml/credit_model.py merges banking features into the unified model input set.
  • backend/src/ml/credit_pipeline_validation.py uses the canonical banking object to determine pass, refer, or decline outcomes.
  • frontend/lib/financial-insights.ts normalizes account payloads for dashboard aggregation.
  • frontend/lib/credit-risk.ts canonicalizes provider names used by UI-side scoring helpers.

Error Handling

  • OpenBankingAdapter.from_connected_accounts returns CanonicalOpenBankingData(provider="unknown") when the input is not a dict or list.
  • OpenBankingAdapter.from_connected_accounts tolerates malformed provider payloads by skipping non-dict account entries and non-numeric amounts.
  • OpenBankingAdapter.from_plaid and from_saltedge preserve canonicalization even when the provider-specific top-level envelope differs.
  • build_balance_features falls back to zero-valued aggregates when balances are missing or empty.
  • normalizeConnectedAccounts drops malformed account entries instead of failing the full normalization pass.
  • normalizeProvider returns null for empty or unmapped provider strings.
  • DataCompletenessReport carries the failure state forward through flags and gate_result instead of throwing during normalization.

Dependencies

  • Backend feature engineering uses pandas and numpy.
  • Canonical schema objects use Python dataclasses, datetime, and typing.
  • Downstream model alignment is shared with backend/src/ml/credit_model.py and backend/src/ml/credit_pipeline_validation.py.
  • Frontend normalization helpers use TypeScript string and collection primitives, plus Next.js application data shapes from the surrounding frontend codebase.
  • Provider alias normalization in frontend/lib/credit-risk.ts shares the same provider vocabulary as the backend adapter layer.

Testing Considerations

The visible tests exercise the following behaviors:

  • Plaid provider detection in OpenBankingAdapter.from_plaid.
  • Salt Edge provider detection in OpenBankingAdapter.from_saltedge.
  • Account count aggregation across provider groups in OpenBankingAdapter.from_connected_accounts.
  • Handling of None payloads and bare list payloads.
  • Category lowercasing for transaction records.
  • Positive USD conversion for transaction amounts.
  • GBP balance conversion using _DEFAULT_FX_TO_USD.
  • Transaction merging by account_id in Plaid payloads.
  • Preservation of institution names in the canonical institutions list.
  • days_covered coverage behavior in canonical banking data construction.

Key Classes Reference

Class Responsibility
credit_canonical_schema.py Canonical banking, bureau, and completeness schemas for scoring inputs.
credit_adapters.py Provider-specific payload normalization into canonical banking data.
open_banking_features.py Banking feature extraction for score generation.
financial-insights.ts Frontend normalization of connected accounts and descriptor text.
credit-risk.ts Provider alias canonicalization for UI-side risk helpers.
credit_model.py Downstream model alignment for open-banking feature columns.
credit_pipeline_validation.py Completeness gating and pass or refer or decline decisions.