Credit Scoring and Pipeline Domain - Feature Engineering, Open Banking Feature Extraction, and Validation Gates
Overview
This domain turns canonical credit inputs and open-banking transaction data into model-ready feature sets, then enforces deterministic validation gates before scoring or training continues. The feature engineering code concentrates on three output shapes: per-customer open-banking feature vectors, training CSVs for those features, and completeness reports that decide whether an application can pass, must be referred, or must be declined.
For the product, this is the control point that prevents incomplete, stale, or inconsistent inputs from reaching the model contract. The validation layer attaches a completeness report back onto the canonical input object, and the feature builders convert raw transactions, balances, and country signals into stable numeric fields that the unified credit model can consume.
Architecture Overview
flowchart TB
subgraph Inputs [Canonical and Banking Inputs]
Canonical[CanonicalCreditInput]
Tx[Transactions DataFrame]
Accounts[Accounts DataFrame]
Country[Country code]
end
subgraph FeatureEngineering [Open Banking Feature Engineering]
OBF[open_banking_features.py]
Vector[build_open_banking_feature_vector]
end
subgraph CsvBuilder [Training CSV Builder]
CSV[build_open_banking_csvs.py]
Templates[CSV templates]
GCS[Google Cloud Storage]
end
subgraph Validation [Pipeline Validation]
CPV[credit_pipeline_validation.py]
Report[DataCompletenessReport]
Gate[GatingResult]
end
subgraph Tests [Regression Coverage]
TPV[test_pipeline_validation.py]
end
Tx --> Vector
Accounts --> Vector
Country --> Vector
Vector --> OBF
Tx --> CSV
Accounts --> CSV
Country --> CSV
CSV --> Templates
CSV --> GCS
Canonical --> CPV
CPV --> Report
Report --> Gate
Gate --> Canonical
TPV --> CPV
TPV --> OBF
TPV --> CSV
Open Banking Feature Engineering
File: backend/src/ml/open_banking_features.py
This module derives open-banking features from transactions, account snapshots, and country context. Its main responsibility is to produce a one-row pandas.DataFrame that can be concatenated into the broader credit feature set.
Public functions
| Method | Description |
|---|---|
build_income_features |
Derives monthly income, stability, recurring income, source diversity, deposit volatility, and minimum monthly income from transaction data. |
build_balance_features |
Summarizes account balance history and cashflow behavior into balance and liquidity features. |
build_country_risk_features |
Maps a country code to macro-risk fields with a fallback profile when the code is not found. |
build_open_banking_feature_vector |
Combines income, balance, spending, stability, and country signals into a single feature row, with optional extra columns appended. |
Derived feature outputs
build_income_features
Returns a Dict[str, float] with these keys:
| Field | Description |
|---|---|
monthly_income |
Mean monthly income across observed income buckets. |
income_variance_6m |
Population variance of the last 6 monthly income buckets. |
recurring_income_3m |
Mean of the last 3 monthly income buckets. |
income_source_count |
Distinct source count from the source column. |
deposit_volatility_score |
Standard deviation divided by mean-plus-one for deposit amounts. |
min_monthly_income |
Minimum observed monthly income bucket. |
Implementation notes tied to the code:
- If a
typecolumn exists, only rows wheretype == "income"are used. - The function groups by calendar month using the
datecolumn. - Empty inputs fall back to
0.0for every numeric output.
build_balance_features
Returns a Dict[str, float] with these keys:
| Field | Description |
|---|---|
avg_daily_balance |
Mean of the historical_balances series. |
min_balance_30d |
Minimum balance across the last 30 balance observations. |
overdraft_count |
Count of negative balance observations. |
bounce_count |
Sum of the account-level bounce_count field. |
cash_buffer_ratio |
Average daily balance divided by outflow-plus-one, guarded against empty data. |
net_cashflow_month |
Total inflow minus total outflow. |
surplus_ratio |
Net monthly cashflow divided by inflow-plus-one. |
Implementation notes tied to the code:
historical_balancesmay be an object-typed series of lists or arrays; the function explodes and casts them before calculating.- The function reads
bounce_count,inflow, andoutflowdirectly from the account frame. - Empty inputs fall back to
0.0.
build_country_risk_features
Returns a Dict[str, float] with these keys:
| Field | Description |
|---|---|
country_code |
Original input country code. |
macro_risk |
Macro-risk score for the country. |
gdp_growth |
GDP growth value for the country. |
inflation |
Inflation value for the country. |
corruption_index |
Corruption index value for the country. |
Implementation notes tied to the code:
- The lookup uses
country_code.upper(). - If the country is not present, the function falls back to the
JPprofile. - The returned mapping includes
country_codeas a string entry.
build_open_banking_feature_vector
[!NOTE]
build_country_risk_featuresis annotated asDict[str, float], but it returnscountry_codeas a string value alongside numeric fields. Any consumer that assumes all values are numeric will mis-handle that mapping.
Returns a one-row pandas.DataFrame assembled from the dictionaries returned by the feature builders.
| Source | Behavior |
|---|---|
build_income_features(tx) |
Adds income-related columns. |
build_balance_features(accounts) |
Adds balance and cashflow columns. |
build_spending_features(tx) |
Adds spending-related columns. |
build_stability_features(tx) |
Adds expense or stability columns. |
build_country_risk_features(country_code) |
Adds corridor-risk columns. |
extra_features |
If provided, concatenated as additional columns after index reset. |
Implementation notes tied to the code:
- The output is always created from a single
feature_dictrow. extra_featuresis concatenated horizontally, not merged by key.- This function is the aggregation point used by runtime feature construction and training workflows.
Feature aggregation flow
sequenceDiagram
participant Caller as Caller
participant OBF as open_banking_features
participant Income as build_income_features
participant Balance as build_balance_features
participant Spending as build_spending_features
participant Stability as build_stability_features
participant Country as build_country_risk_features
Caller->>OBF: build_open_banking_feature_vector
OBF->>Income: build_income_features
OBF->>Balance: build_balance_features
OBF->>Spending: build_spending_features
OBF->>Stability: build_stability_features
OBF->>Country: build_country_risk_features
Country-->>OBF: country metrics
Stability-->>OBF: stability metrics
Spending-->>OBF: spending metrics
Balance-->>OBF: balance metrics
Income-->>OBF: income metrics
OBF-->>Caller: one row DataFrame
Open Banking CSV Builder
File: backend/src/ml/build_open_banking_csvs.py
This utility mirrors the feature builders into CSV artifacts for training and backfill workflows, and it can upload generated CSVs into Google Cloud Storage. The module also includes a __main__ path that initializes empty template files with the expected columns.
Public functions
| Method | Description |
|---|---|
build_income_features |
Delegates to obf.build_income_features(tx_df) for CSV generation. |
build_cashflow_features |
Builds cashflow-oriented balance and transaction features directly from transaction and account frames. |
build_expense_stability |
Delegates to obf.build_stability_features(tx_df). |
build_behavior_risk |
Delegates to obf.build_spending_features(tx_df). |
build_country_risk |
Delegates to obf.build_country_risk_features(country_code). |
upload_open_banking_csvs_to_gcs |
Uploads local *.csv files from a directory into a GCS bucket path. |
CSV feature outputs
build_income_features
Returns the same income feature keys as the runtime feature builder:
| Field |
|---|
monthly_income |
income_variance_6m |
recurring_income_3m |
income_source_count |
deposit_volatility_score |
min_monthly_income |
build_cashflow_features
Returns a Dict[str, float] with these keys:
| Field | Description |
|---|---|
avg_daily_balance |
Mean balance across the historical balance series. |
min_balance_30d |
Minimum balance over the last 30 balance points. |
overdraft_count |
Count of negative balances. |
bounce_count |
Sum of account-level bounce counts. |
cash_buffer_ratio |
Average daily balance divided by outflow-plus-one. |
net_cashflow_month |
Monthly inflow minus monthly outflow. |
surplus_ratio |
Net cashflow divided by inflow-plus-one. |
Implementation notes tied to the code:
historical_balancescan be list-like inside an object series and is exploded before computation.inflowis computed from transaction rows whereamount > 0.outflowis computed from transaction rows whereamount < 0.
build_expense_stability
[!NOTE]
build_balance_featuresinopen_banking_features.pyandbuild_cashflow_featuresinbuild_open_banking_csvs.pydo not computecash_buffer_ratiothe same way when outflow is zero. The runtime helper returns0.0in that case, while the CSV builder divides byoutflow + 1, which producesavg_daily_balance. That creates a feature-parity mismatch between training and runtime paths.
Delegates to obf.build_stability_features(tx_df). The wrapper preserves the CSV builder contract while centralizing the stability feature logic in open_banking_features.py.
build_behavior_risk
Delegates to obf.build_spending_features(tx_df). The wrapper is used for the behavior-risk CSV output.
build_country_risk
Delegates to obf.build_country_risk_features(country_code) and feeds the country-risk CSV output.
Generated CSV templates
When run as a script, the module creates missing CSV templates under data/open_banking/ with these columns:
| File | Columns |
|---|---|
income_features.csv |
customer_id, monthly_income, income_variance_6m, recurring_income_3m, income_source_count, deposit_volatility_score, min_monthly_income |
cashflow_features.csv |
customer_id, avg_daily_balance, min_balance_30d, overdraft_count, bounce_count, cash_buffer_ratio, net_cashflow_month, surplus_ratio |
expense_stability.csv |
customer_id, rent_stability, utility_stability, subscription_count, loan_payment_consistency, fixed_expense_ratio |
behavior_risk.csv |
customer_id, gambling_ratio, atm_frequency, late_night_spending_ratio, spending_spike_events, unknown_fund_sources |
country_risk.csv |
country_code, macro_risk, gdp_growth, inflation, corruption_index |
GCS upload flow
sequenceDiagram
participant Caller as Caller
participant Builder as upload_open_banking_csvs_to_gcs
participant Client as storage Client
participant Bucket as GCS Bucket
participant Blob as GCS Blob
Caller->>Builder: upload_open_banking_csvs_to_gcs
Builder->>Client: create client
Builder->>Bucket: select bucket
loop each csv file in local_dir
Builder->>Blob: upload_from_filename
end
Blob-->>Builder: upload complete
Builder-->>Caller: done
Pipeline Validation Gates
File: backend/src/ml/credit_pipeline_validation.py
This module decides whether an application can be scored, must be referred, or must be declined. It computes a completeness report from CanonicalCreditInput, classifies the result into blocking and warning flags, and attaches the report back onto the canonical input object for downstream use.
Configuration constants
| Constant | Default | Meaning |
|---|---|---|
MIN_DAYS_COVERED |
60 |
Minimum banking history required before the history gate is satisfied. |
BUREAU_MAX_AGE_DAYS |
90 |
Maximum allowed bureau report age before the report is stale. |
MIN_COMPLETENESS_SCORE_TO_PASS |
0.50 |
Minimum completeness score required when no decline or refer flags are present. |
_BUREAU_REQUIRED_PRODUCTS |
high_ticket_loan |
Product types that require bureau data, if configured absent. |
Flag sets
| Set | Members |
|---|---|
_DECLINE_FLAGS |
KYC_NOT_PASSED, FRAUD_RISK_DETECTED, AML_RISK_DETECTED |
_REFER_FLAGS |
BUREAU_REQUIRED_BUT_ABSENT, NO_BANKING_DATA, INSUFFICIENT_HISTORY |
Public functions
| Method | Description |
|---|---|
compute_completeness |
Builds the completeness report, assigns flags, computes the score, and determines the gate result. |
gate |
Calls compute_completeness, stores the report on canonical.data_completeness, and returns a GatingResult. |
Core gating behavior
_gate_from_flags
This helper applies the gate priority in a fixed order:
- Any decline flag returns
decline. - Any refer flag returns
refer. - If the score is below
MIN_COMPLETENESS_SCORE_TO_PASS, returnsrefer. - Otherwise returns
pass.
compute_completeness
This function builds the DataCompletenessReport from the canonical payload and calculates the weighted completeness score.
| Signal | Weight | Contribution Rule |
|---|---|---|
kyc_passed |
0.20 |
Counts if KYC is true. |
banking_available |
0.20 |
Counts if banking data is present. |
days_covered_ok |
0.15 |
Counts if the banking history threshold is met. |
salary_detected |
0.10 |
Counts if salary-like transactions are detected. |
income_verified |
0.10 |
Counts if income is established by salary, bureau income, or stated income. |
bureau_available |
0.15 |
Counts if bureau data is present. |
bureau_recency_ok |
0.10 |
Counts when bureau is available and fresh; treated as satisfied when bureau is absent. |
It also populates flags for specific rule outcomes:
| Flag | Trigger |
|---|---|
KYC_NOT_PASSED |
kyc_passed is false. |
FRAUD_RISK_DETECTED |
Bureau fraud flag is set. |
AML_RISK_DETECTED |
Bureau AML flag is set. |
NO_BANKING_DATA |
Banking data is missing. |
INSUFFICIENT_HISTORY |
days_covered is below MIN_DAYS_COVERED. |
BUREAU_REPORT_STALE |
Bureau recency exceeds BUREAU_MAX_AGE_DAYS. |
INCOME_UNVERIFIED |
Neither salary, bureau income, nor stated income is available. |
INCOME_STATED_ONLY |
Stated income exists, but salary and bureau income do not. |
BUREAU_REQUIRED_BUT_ABSENT |
The product is in _BUREAU_REQUIRED_PRODUCTS and bureau data is missing. |
Implementation notes tied to the code:
- Salary detection is driven by transaction categories such as
salaryandpayroll. income_verifiedbecomes true if salary is detected, if bureau estimated income is present, or if stated income is greater than zero.- The completeness score is rounded to four decimal places.
- The score is computed even when some inputs are missing; the weighting logic determines the result.
gate
gate() is the stateful wrapper around compute_completeness().
| Output field | Description |
|---|---|
gate_result |
The final pass, refer, or decline outcome. |
blocking_flags |
Flags that belong to either the decline or refer sets. |
warn_flags |
Flags that are neither blocking decline nor refer conditions. |
completeness_score |
The calculated completeness score. |
The function also mutates the input canonical object by assigning canonical.data_completeness = report.
Validation flow
sequenceDiagram
participant Caller as Caller
participant CPV as credit_pipeline_validation
participant Report as DataCompletenessReport
participant Canonical as CanonicalCreditInput
Caller->>CPV: gate
CPV->>CPV: compute_completeness
CPV->>CPV: _gate_from_flags
CPV-->>Report: populate flags and score
CPV-->>Canonical: assign data_completeness
CPV-->>Caller: GatingResult
State management
The only explicit state mutation in this section is the assignment performed by gate():
canonical.data_completenessis set to the computed report.- The report is then reused by downstream feature engineering and orchestration code without recomputation.
Validation Test Coverage
File: backend/tests/ml/test_pipeline_validation.py
The test module exercises the gate priority, score behavior, bureau freshness logic, salary detection, and the side effect that stores the report on the canonical input.
Test helpers
| Helper | Purpose |
|---|---|
_tx |
Builds a CanonicalTransaction with a timestamp, amount, direction, category, and counterparty hash. |
_banking |
Builds a CanonicalOpenBankingData payload with account count, coverage days, and transactions. |
_bureau |
Builds a CanonicalBureauData payload with score, recency, and risk flags. |
_full_canonical |
Builds a CanonicalCreditInput that passes the gates by default, then applies overrides. |
Test classes and methods
TestDeclineGates
| Method | What it verifies |
|---|---|
test_kyc_not_passed_gives_decline |
A false KYC flag yields decline and KYC_NOT_PASSED. |
test_fraud_flag_gives_decline |
Bureau fraud risk yields decline and FRAUD_RISK_DETECTED. |
test_aml_flag_gives_decline |
Bureau AML risk yields decline and AML_RISK_DETECTED. |
test_decline_takes_priority_over_refer |
Decline flags win even when banking is also missing. |
TestReferGates
| Method | What it verifies |
|---|---|
test_no_banking_data_gives_refer |
Missing banking data yields refer and NO_BANKING_DATA. |
test_insufficient_history_gives_refer |
Banking coverage below MIN_DAYS_COVERED yields refer and INSUFFICIENT_HISTORY. |
test_exactly_min_days_covered_does_not_refer_on_history |
Exactly the threshold does not trigger INSUFFICIENT_HISTORY. |
test_high_ticket_loan_without_bureau_gives_refer |
high_ticket_loan without bureau data yields refer and BUREAU_REQUIRED_BUT_ABSENT. |
test_regular_product_without_bureau_does_not_require_bureau |
Non-required products do not add the bureau-required flag. |
TestPassGate
| Method | What it verifies |
|---|---|
test_full_data_passes |
The default canonical payload passes. |
test_pass_without_bureau_for_rental |
A rental application can still pass without bureau data if the completeness score stays above threshold. |
TestCompletenessScore
| Method | What it verifies |
|---|---|
test_full_data_gives_score_above_threshold |
The default payload reaches at least 0.50. |
test_score_is_near_zero_when_nothing_available |
An empty payload collapses to the minimal score contribution of 0.10. |
test_score_bounded_between_0_and_1 |
The score stays within bounds. |
test_adding_bureau_increases_score |
Adding bureau data increases completeness. |
TestBureauAvailability
| Method | What it verifies |
|---|---|
test_bureau_available_true_when_score_present |
Positive bureau score marks bureau as available. |
test_bureau_available_false_when_score_zero |
Zero bureau score marks bureau as unavailable. |
test_bureau_available_false_when_none |
Missing bureau data marks bureau as unavailable. |
test_stale_bureau_adds_flag |
A stale bureau report sets BUREAU_REPORT_STALE and clears bureau_recency_ok. |
test_fresh_bureau_no_stale_flag |
Fresh bureau data does not add the stale flag and keeps bureau_recency_ok true. |
TestIncomeDetection
| Method | What it verifies |
|---|---|
test_salary_category_detected |
A salary transaction category sets salary detection. |
test_payroll_category_detected |
A payroll category also sets salary detection. |
test_non_income_category_not_detected_as_salary |
Non-income categories do not trigger salary detection. |
test_income_verified_via_stated_income |
Stated income can satisfy income verification and adds INCOME_STATED_ONLY when salary and bureau income are absent. |
test_income_verified_via_bureau_income |
Bureau estimated income can satisfy income verification. |
test_income_unverified_when_nothing_present |
Missing salary, bureau income, and stated income yields INCOME_UNVERIFIED. |
TestGateFunction
| Method | What it verifies |
|---|---|
test_gate_attaches_report_to_canonical |
gate() writes the report back to canonical.data_completeness. |
test_gate_separates_blocking_and_warn_flags |
Blocking flags are split from warning flags. |
test_gate_result_decline_returned_correctly |
gate() returns decline for a blocked canonical payload. |
test_gate_result_pass_returned_correctly |
gate() returns pass for a clean canonical payload. |
test_gate_completeness_score_populated |
gate() returns a result with a populated completeness score. |
Testing focus
| Area | Coverage |
|---|---|
| Gate priority | Decline beats refer, and refer beats score-based pass. |
| Completeness scoring | Weighted contributions, bounds, and threshold behavior. |
| Bureau handling | Availability, staleness, and product-specific requirements. |
| Income detection | Salary categories, stated income, bureau income, and unverified cases. |
| State propagation | The report is attached to the canonical input object. |
Error Handling and Deterministic Outcomes
The code in this section uses explicit flagging rather than hidden fallback behavior:
compute_completeness()records missing or stale conditions inflags.gate()returns structuredblocking_flagsandwarn_flagsrather than flattening the outcome into a single string.upload_open_banking_csvs_to_gcs()iterates through local CSV files and delegates upload directly to the GCS client.- The CSV template initializer creates files only when they do not already exist.
Dependencies
| File | Dependency | Role |
|---|---|---|
open_banking_features.py |
numpy, pandas |
Numeric aggregation and DataFrame construction. |
build_open_banking_csvs.py |
pandas, google.cloud.storage, Path |
CSV creation and upload to GCS. |
credit_pipeline_validation.py |
os, dataclasses, typing, src.ml.credit_canonical_schema |
Threshold configuration, structured gate results, and canonical input inspection. |
test_pipeline_validation.py |
pytest, datetime, src.ml.credit_canonical_schema, src.ml.credit_pipeline_validation |
Regression coverage for gating behavior and score computation. |
Key Classes Reference
| Class | Responsibility |
|---|---|
open_banking_features.py |
Derives open-banking feature rows from transactions, balances, and country context. |
build_open_banking_csvs.py |
Builds feature CSVs and uploads them to Google Cloud Storage. |
credit_pipeline_validation.py |
Computes completeness, applies gating rules, and attaches the report to canonical input. |
test_pipeline_validation.py |
Verifies decline, refer, pass, scoring, and income detection behavior. |