Credit Scoring and Pipeline Domain - Feature Engineering, Open Banking Feature Extraction, and Validation Gates

Overview

This domain turns canonical credit inputs and open-banking transaction data into model-ready feature sets, then enforces deterministic validation gates before scoring or training continues. The feature engineering code concentrates on three output shapes: per-customer open-banking feature vectors, training CSVs for those features, and completeness reports that decide whether an application can pass, must be referred, or must be declined.

For the product, this is the control point that prevents incomplete, stale, or inconsistent inputs from reaching the model contract. The validation layer attaches a completeness report back onto the canonical input object, and the feature builders convert raw transactions, balances, and country signals into stable numeric fields that the unified credit model can consume.

Architecture Overview

flowchart TB
    subgraph Inputs [Canonical and Banking Inputs]
        Canonical[CanonicalCreditInput]
        Tx[Transactions DataFrame]
        Accounts[Accounts DataFrame]
        Country[Country code]
    end

    subgraph FeatureEngineering [Open Banking Feature Engineering]
        OBF[open_banking_features.py]
        Vector[build_open_banking_feature_vector]
    end

    subgraph CsvBuilder [Training CSV Builder]
        CSV[build_open_banking_csvs.py]
        Templates[CSV templates]
        GCS[Google Cloud Storage]
    end

    subgraph Validation [Pipeline Validation]
        CPV[credit_pipeline_validation.py]
        Report[DataCompletenessReport]
        Gate[GatingResult]
    end

    subgraph Tests [Regression Coverage]
        TPV[test_pipeline_validation.py]
    end

    Tx --> Vector
    Accounts --> Vector
    Country --> Vector
    Vector --> OBF

    Tx --> CSV
    Accounts --> CSV
    Country --> CSV
    CSV --> Templates
    CSV --> GCS

    Canonical --> CPV
    CPV --> Report
    Report --> Gate
    Gate --> Canonical

    TPV --> CPV
    TPV --> OBF
    TPV --> CSV

Open Banking Feature Engineering

File: backend/src/ml/open_banking_features.py

This module derives open-banking features from transactions, account snapshots, and country context. Its main responsibility is to produce a one-row pandas.DataFrame that can be concatenated into the broader credit feature set.

Public functions

Method Description
build_income_features Derives monthly income, stability, recurring income, source diversity, deposit volatility, and minimum monthly income from transaction data.
build_balance_features Summarizes account balance history and cashflow behavior into balance and liquidity features.
build_country_risk_features Maps a country code to macro-risk fields with a fallback profile when the code is not found.
build_open_banking_feature_vector Combines income, balance, spending, stability, and country signals into a single feature row, with optional extra columns appended.

Derived feature outputs

build_income_features

Returns a Dict[str, float] with these keys:

Field Description
monthly_income Mean monthly income across observed income buckets.
income_variance_6m Population variance of the last 6 monthly income buckets.
recurring_income_3m Mean of the last 3 monthly income buckets.
income_source_count Distinct source count from the source column.
deposit_volatility_score Standard deviation divided by mean-plus-one for deposit amounts.
min_monthly_income Minimum observed monthly income bucket.

Implementation notes tied to the code:

  • If a type column exists, only rows where type == "income" are used.
  • The function groups by calendar month using the date column.
  • Empty inputs fall back to 0.0 for every numeric output.

build_balance_features

Returns a Dict[str, float] with these keys:

Field Description
avg_daily_balance Mean of the historical_balances series.
min_balance_30d Minimum balance across the last 30 balance observations.
overdraft_count Count of negative balance observations.
bounce_count Sum of the account-level bounce_count field.
cash_buffer_ratio Average daily balance divided by outflow-plus-one, guarded against empty data.
net_cashflow_month Total inflow minus total outflow.
surplus_ratio Net monthly cashflow divided by inflow-plus-one.

Implementation notes tied to the code:

  • historical_balances may be an object-typed series of lists or arrays; the function explodes and casts them before calculating.
  • The function reads bounce_count, inflow, and outflow directly from the account frame.
  • Empty inputs fall back to 0.0.

build_country_risk_features

Returns a Dict[str, float] with these keys:

Field Description
country_code Original input country code.
macro_risk Macro-risk score for the country.
gdp_growth GDP growth value for the country.
inflation Inflation value for the country.
corruption_index Corruption index value for the country.

Implementation notes tied to the code:

  • The lookup uses country_code.upper().
  • If the country is not present, the function falls back to the JP profile.
  • The returned mapping includes country_code as a string entry.

build_open_banking_feature_vector

[!NOTE] build_country_risk_features is annotated as Dict[str, float], but it returns country_code as a string value alongside numeric fields. Any consumer that assumes all values are numeric will mis-handle that mapping.

Returns a one-row pandas.DataFrame assembled from the dictionaries returned by the feature builders.

Source Behavior
build_income_features(tx) Adds income-related columns.
build_balance_features(accounts) Adds balance and cashflow columns.
build_spending_features(tx) Adds spending-related columns.
build_stability_features(tx) Adds expense or stability columns.
build_country_risk_features(country_code) Adds corridor-risk columns.
extra_features If provided, concatenated as additional columns after index reset.

Implementation notes tied to the code:

  • The output is always created from a single feature_dict row.
  • extra_features is concatenated horizontally, not merged by key.
  • This function is the aggregation point used by runtime feature construction and training workflows.

Feature aggregation flow

sequenceDiagram
    participant Caller as Caller
    participant OBF as open_banking_features
    participant Income as build_income_features
    participant Balance as build_balance_features
    participant Spending as build_spending_features
    participant Stability as build_stability_features
    participant Country as build_country_risk_features

    Caller->>OBF: build_open_banking_feature_vector
    OBF->>Income: build_income_features
    OBF->>Balance: build_balance_features
    OBF->>Spending: build_spending_features
    OBF->>Stability: build_stability_features
    OBF->>Country: build_country_risk_features
    Country-->>OBF: country metrics
    Stability-->>OBF: stability metrics
    Spending-->>OBF: spending metrics
    Balance-->>OBF: balance metrics
    Income-->>OBF: income metrics
    OBF-->>Caller: one row DataFrame

Open Banking CSV Builder

File: backend/src/ml/build_open_banking_csvs.py

This utility mirrors the feature builders into CSV artifacts for training and backfill workflows, and it can upload generated CSVs into Google Cloud Storage. The module also includes a __main__ path that initializes empty template files with the expected columns.

Public functions

Method Description
build_income_features Delegates to obf.build_income_features(tx_df) for CSV generation.
build_cashflow_features Builds cashflow-oriented balance and transaction features directly from transaction and account frames.
build_expense_stability Delegates to obf.build_stability_features(tx_df).
build_behavior_risk Delegates to obf.build_spending_features(tx_df).
build_country_risk Delegates to obf.build_country_risk_features(country_code).
upload_open_banking_csvs_to_gcs Uploads local *.csv files from a directory into a GCS bucket path.

CSV feature outputs

build_income_features

Returns the same income feature keys as the runtime feature builder:

Field
monthly_income
income_variance_6m
recurring_income_3m
income_source_count
deposit_volatility_score
min_monthly_income

build_cashflow_features

Returns a Dict[str, float] with these keys:

Field Description
avg_daily_balance Mean balance across the historical balance series.
min_balance_30d Minimum balance over the last 30 balance points.
overdraft_count Count of negative balances.
bounce_count Sum of account-level bounce counts.
cash_buffer_ratio Average daily balance divided by outflow-plus-one.
net_cashflow_month Monthly inflow minus monthly outflow.
surplus_ratio Net cashflow divided by inflow-plus-one.

Implementation notes tied to the code:

  • historical_balances can be list-like inside an object series and is exploded before computation.
  • inflow is computed from transaction rows where amount > 0.
  • outflow is computed from transaction rows where amount < 0.

build_expense_stability

[!NOTE] build_balance_features in open_banking_features.py and build_cashflow_features in build_open_banking_csvs.py do not compute cash_buffer_ratio the same way when outflow is zero. The runtime helper returns 0.0 in that case, while the CSV builder divides by outflow + 1, which produces avg_daily_balance. That creates a feature-parity mismatch between training and runtime paths.

Delegates to obf.build_stability_features(tx_df). The wrapper preserves the CSV builder contract while centralizing the stability feature logic in open_banking_features.py.

build_behavior_risk

Delegates to obf.build_spending_features(tx_df). The wrapper is used for the behavior-risk CSV output.

build_country_risk

Delegates to obf.build_country_risk_features(country_code) and feeds the country-risk CSV output.

Generated CSV templates

When run as a script, the module creates missing CSV templates under data/open_banking/ with these columns:

File Columns
income_features.csv customer_id, monthly_income, income_variance_6m, recurring_income_3m, income_source_count, deposit_volatility_score, min_monthly_income
cashflow_features.csv customer_id, avg_daily_balance, min_balance_30d, overdraft_count, bounce_count, cash_buffer_ratio, net_cashflow_month, surplus_ratio
expense_stability.csv customer_id, rent_stability, utility_stability, subscription_count, loan_payment_consistency, fixed_expense_ratio
behavior_risk.csv customer_id, gambling_ratio, atm_frequency, late_night_spending_ratio, spending_spike_events, unknown_fund_sources
country_risk.csv country_code, macro_risk, gdp_growth, inflation, corruption_index

GCS upload flow

sequenceDiagram
    participant Caller as Caller
    participant Builder as upload_open_banking_csvs_to_gcs
    participant Client as storage Client
    participant Bucket as GCS Bucket
    participant Blob as GCS Blob

    Caller->>Builder: upload_open_banking_csvs_to_gcs
    Builder->>Client: create client
    Builder->>Bucket: select bucket
    loop each csv file in local_dir
        Builder->>Blob: upload_from_filename
    end
    Blob-->>Builder: upload complete
    Builder-->>Caller: done

Pipeline Validation Gates

File: backend/src/ml/credit_pipeline_validation.py

This module decides whether an application can be scored, must be referred, or must be declined. It computes a completeness report from CanonicalCreditInput, classifies the result into blocking and warning flags, and attaches the report back onto the canonical input object for downstream use.

Configuration constants

Constant Default Meaning
MIN_DAYS_COVERED 60 Minimum banking history required before the history gate is satisfied.
BUREAU_MAX_AGE_DAYS 90 Maximum allowed bureau report age before the report is stale.
MIN_COMPLETENESS_SCORE_TO_PASS 0.50 Minimum completeness score required when no decline or refer flags are present.
_BUREAU_REQUIRED_PRODUCTS high_ticket_loan Product types that require bureau data, if configured absent.

Flag sets

Set Members
_DECLINE_FLAGS KYC_NOT_PASSED, FRAUD_RISK_DETECTED, AML_RISK_DETECTED
_REFER_FLAGS BUREAU_REQUIRED_BUT_ABSENT, NO_BANKING_DATA, INSUFFICIENT_HISTORY

Public functions

Method Description
compute_completeness Builds the completeness report, assigns flags, computes the score, and determines the gate result.
gate Calls compute_completeness, stores the report on canonical.data_completeness, and returns a GatingResult.

Core gating behavior

_gate_from_flags

This helper applies the gate priority in a fixed order:

  1. Any decline flag returns decline.
  2. Any refer flag returns refer.
  3. If the score is below MIN_COMPLETENESS_SCORE_TO_PASS, returns refer.
  4. Otherwise returns pass.

compute_completeness

This function builds the DataCompletenessReport from the canonical payload and calculates the weighted completeness score.

Signal Weight Contribution Rule
kyc_passed 0.20 Counts if KYC is true.
banking_available 0.20 Counts if banking data is present.
days_covered_ok 0.15 Counts if the banking history threshold is met.
salary_detected 0.10 Counts if salary-like transactions are detected.
income_verified 0.10 Counts if income is established by salary, bureau income, or stated income.
bureau_available 0.15 Counts if bureau data is present.
bureau_recency_ok 0.10 Counts when bureau is available and fresh; treated as satisfied when bureau is absent.

It also populates flags for specific rule outcomes:

Flag Trigger
KYC_NOT_PASSED kyc_passed is false.
FRAUD_RISK_DETECTED Bureau fraud flag is set.
AML_RISK_DETECTED Bureau AML flag is set.
NO_BANKING_DATA Banking data is missing.
INSUFFICIENT_HISTORY days_covered is below MIN_DAYS_COVERED.
BUREAU_REPORT_STALE Bureau recency exceeds BUREAU_MAX_AGE_DAYS.
INCOME_UNVERIFIED Neither salary, bureau income, nor stated income is available.
INCOME_STATED_ONLY Stated income exists, but salary and bureau income do not.
BUREAU_REQUIRED_BUT_ABSENT The product is in _BUREAU_REQUIRED_PRODUCTS and bureau data is missing.

Implementation notes tied to the code:

  • Salary detection is driven by transaction categories such as salary and payroll.
  • income_verified becomes true if salary is detected, if bureau estimated income is present, or if stated income is greater than zero.
  • The completeness score is rounded to four decimal places.
  • The score is computed even when some inputs are missing; the weighting logic determines the result.

gate

gate() is the stateful wrapper around compute_completeness().

Output field Description
gate_result The final pass, refer, or decline outcome.
blocking_flags Flags that belong to either the decline or refer sets.
warn_flags Flags that are neither blocking decline nor refer conditions.
completeness_score The calculated completeness score.

The function also mutates the input canonical object by assigning canonical.data_completeness = report.

Validation flow

sequenceDiagram
    participant Caller as Caller
    participant CPV as credit_pipeline_validation
    participant Report as DataCompletenessReport
    participant Canonical as CanonicalCreditInput

    Caller->>CPV: gate
    CPV->>CPV: compute_completeness
    CPV->>CPV: _gate_from_flags
    CPV-->>Report: populate flags and score
    CPV-->>Canonical: assign data_completeness
    CPV-->>Caller: GatingResult

State management

The only explicit state mutation in this section is the assignment performed by gate():

  • canonical.data_completeness is set to the computed report.
  • The report is then reused by downstream feature engineering and orchestration code without recomputation.

Validation Test Coverage

File: backend/tests/ml/test_pipeline_validation.py

The test module exercises the gate priority, score behavior, bureau freshness logic, salary detection, and the side effect that stores the report on the canonical input.

Test helpers

Helper Purpose
_tx Builds a CanonicalTransaction with a timestamp, amount, direction, category, and counterparty hash.
_banking Builds a CanonicalOpenBankingData payload with account count, coverage days, and transactions.
_bureau Builds a CanonicalBureauData payload with score, recency, and risk flags.
_full_canonical Builds a CanonicalCreditInput that passes the gates by default, then applies overrides.

Test classes and methods

TestDeclineGates

Method What it verifies
test_kyc_not_passed_gives_decline A false KYC flag yields decline and KYC_NOT_PASSED.
test_fraud_flag_gives_decline Bureau fraud risk yields decline and FRAUD_RISK_DETECTED.
test_aml_flag_gives_decline Bureau AML risk yields decline and AML_RISK_DETECTED.
test_decline_takes_priority_over_refer Decline flags win even when banking is also missing.

TestReferGates

Method What it verifies
test_no_banking_data_gives_refer Missing banking data yields refer and NO_BANKING_DATA.
test_insufficient_history_gives_refer Banking coverage below MIN_DAYS_COVERED yields refer and INSUFFICIENT_HISTORY.
test_exactly_min_days_covered_does_not_refer_on_history Exactly the threshold does not trigger INSUFFICIENT_HISTORY.
test_high_ticket_loan_without_bureau_gives_refer high_ticket_loan without bureau data yields refer and BUREAU_REQUIRED_BUT_ABSENT.
test_regular_product_without_bureau_does_not_require_bureau Non-required products do not add the bureau-required flag.

TestPassGate

Method What it verifies
test_full_data_passes The default canonical payload passes.
test_pass_without_bureau_for_rental A rental application can still pass without bureau data if the completeness score stays above threshold.

TestCompletenessScore

Method What it verifies
test_full_data_gives_score_above_threshold The default payload reaches at least 0.50.
test_score_is_near_zero_when_nothing_available An empty payload collapses to the minimal score contribution of 0.10.
test_score_bounded_between_0_and_1 The score stays within bounds.
test_adding_bureau_increases_score Adding bureau data increases completeness.

TestBureauAvailability

Method What it verifies
test_bureau_available_true_when_score_present Positive bureau score marks bureau as available.
test_bureau_available_false_when_score_zero Zero bureau score marks bureau as unavailable.
test_bureau_available_false_when_none Missing bureau data marks bureau as unavailable.
test_stale_bureau_adds_flag A stale bureau report sets BUREAU_REPORT_STALE and clears bureau_recency_ok.
test_fresh_bureau_no_stale_flag Fresh bureau data does not add the stale flag and keeps bureau_recency_ok true.

TestIncomeDetection

Method What it verifies
test_salary_category_detected A salary transaction category sets salary detection.
test_payroll_category_detected A payroll category also sets salary detection.
test_non_income_category_not_detected_as_salary Non-income categories do not trigger salary detection.
test_income_verified_via_stated_income Stated income can satisfy income verification and adds INCOME_STATED_ONLY when salary and bureau income are absent.
test_income_verified_via_bureau_income Bureau estimated income can satisfy income verification.
test_income_unverified_when_nothing_present Missing salary, bureau income, and stated income yields INCOME_UNVERIFIED.

TestGateFunction

Method What it verifies
test_gate_attaches_report_to_canonical gate() writes the report back to canonical.data_completeness.
test_gate_separates_blocking_and_warn_flags Blocking flags are split from warning flags.
test_gate_result_decline_returned_correctly gate() returns decline for a blocked canonical payload.
test_gate_result_pass_returned_correctly gate() returns pass for a clean canonical payload.
test_gate_completeness_score_populated gate() returns a result with a populated completeness score.

Testing focus

Area Coverage
Gate priority Decline beats refer, and refer beats score-based pass.
Completeness scoring Weighted contributions, bounds, and threshold behavior.
Bureau handling Availability, staleness, and product-specific requirements.
Income detection Salary categories, stated income, bureau income, and unverified cases.
State propagation The report is attached to the canonical input object.

Error Handling and Deterministic Outcomes

The code in this section uses explicit flagging rather than hidden fallback behavior:

  • compute_completeness() records missing or stale conditions in flags.
  • gate() returns structured blocking_flags and warn_flags rather than flattening the outcome into a single string.
  • upload_open_banking_csvs_to_gcs() iterates through local CSV files and delegates upload directly to the GCS client.
  • The CSV template initializer creates files only when they do not already exist.

Dependencies

File Dependency Role
open_banking_features.py numpy, pandas Numeric aggregation and DataFrame construction.
build_open_banking_csvs.py pandas, google.cloud.storage, Path CSV creation and upload to GCS.
credit_pipeline_validation.py os, dataclasses, typing, src.ml.credit_canonical_schema Threshold configuration, structured gate results, and canonical input inspection.
test_pipeline_validation.py pytest, datetime, src.ml.credit_canonical_schema, src.ml.credit_pipeline_validation Regression coverage for gating behavior and score computation.

Key Classes Reference

Class Responsibility
open_banking_features.py Derives open-banking feature rows from transactions, balances, and country context.
build_open_banking_csvs.py Builds feature CSVs and uploads them to Google Cloud Storage.
credit_pipeline_validation.py Computes completeness, applies gating rules, and attaches the report to canonical input.
test_pipeline_validation.py Verifies decline, refer, pass, scoring, and income detection behavior.