Credit Scoring and Pipeline Domain - Bureau Ingestion, Uploads, and Batch Processing
Overview
This feature set is the bureau-data entry point for the credit-scoring pipeline. It covers three ingestion paths that feed the same downstream scoring model: signed manual uploads, Equifax instant pulls, and TransUnion batch files.
The implementation keeps each bureau source separate at ingress, then normalizes them into canonical payloads or canonical bureau records before scoring. That normalization is what lets the broader pipeline treat manual uploads, instant reports, and batch rows as comparable evidence for completeness checks, feature generation, and risk scoring.
Architecture Overview
flowchart TB
subgraph UploadIngress [Manual Upload Ingestion]
UploadSessionCreate[create_signed_upload_session]
ObjectStorage[Object storage blob]
UploadFinalize[finalize_upload_session]
OCRMap[map_ocr_to_canonical_credit]
ModelAdapter[build_credit_model_adapter_features]
end
subgraph EquifaxFlow [Equifax Instant Intake]
EquifaxEndpoint[POST api equifax instant intake]
EquifaxUseCase[perform_equifax_instant_intake]
EquifaxClient[EquifaxClient]
EquifaxCleaner[preprocess_equifax_data]
EquifaxSummary[_build_summary]
end
subgraph TransUnionFlow [TransUnion Batch Processing]
TransUnionEndpoint[POST api transunion batch ingestions]
BatchUseCase[process_transunion_batch]
BatchParser[parse_line]
BatchRepo[TransUnionBatchRepository]
BatchSummary[_build_summary]
end
subgraph JapanFlow [Japan Staff Pull]
StaffUseCase[perform_staff_credit_pull]
ConsentRepo[StaffCreditConsentRepository]
PullLogRepo[StaffCreditPullLogRepository]
BureauClient[Bureau client]
end
subgraph ScoringBridge [Canonical Scoring Bridge]
BureauAdapter[BureauAdapter]
CanonicalData[CanonicalBureauData]
PipelineGate[credit_pipeline_validation]
RuntimeScorer[credit_score_runtime]
CreditModel[credit_model]
end
UploadSessionCreate --> ObjectStorage
ObjectStorage --> UploadFinalize
UploadFinalize --> OCRMap
OCRMap --> ModelAdapter
ModelAdapter --> CanonicalData
EquifaxEndpoint --> EquifaxUseCase
EquifaxUseCase --> EquifaxClient
EquifaxUseCase --> EquifaxCleaner
EquifaxUseCase --> EquifaxSummary
EquifaxUseCase --> CanonicalData
TransUnionEndpoint --> BatchUseCase
BatchUseCase --> BatchParser
BatchUseCase --> BatchRepo
BatchUseCase --> BatchSummary
BatchUseCase --> BureauAdapter
BureauAdapter --> CanonicalData
StaffUseCase --> ConsentRepo
StaffUseCase --> PullLogRepo
StaffUseCase --> BureauClient
StaffUseCase --> CanonicalData
CanonicalData --> PipelineGate
PipelineGate --> RuntimeScorer
RuntimeScorer --> CreditModel
Upload Session Lifecycle
File paths: backend/src/usecase/credit_upload.py, backend/tests/usecase/test_credit_upload.py
Manual bureau uploads enter the platform through a signed upload session. The session records the declared bureau, country, object path, file size, and content type before any file is accepted.
Upload Session Abstraction
The upload path uses an UploadSession record to bind the user, bureau, and exact storage object that will hold the document.
| Property | Type | Description |
|---|---|---|
upload_session_id |
str |
Unique upload-session identifier. |
user_id |
str |
Owner of the upload session. |
bureau |
str |
Bureau label used in the object path and later canonical metadata. |
country |
str |
Country code recorded with the session. |
object_path |
str |
Storage path for the uploaded file. |
content_type |
str |
Declared MIME type used for validation. |
file_size_bytes |
int |
Declared file size limit for the upload. |
expires_at |
datetime |
Session expiration timestamp. |
The signed-session response exposes the storage details needed by the uploader.
| Property | Type | Description |
|---|---|---|
upload_session_id |
str |
Session identifier returned to the caller. |
expires_at |
datetime |
Upload window expiration. |
upload_url |
str |
Signed PUT URL for object storage. |
object_path |
str |
Final object path to upload to. |
content_type |
str |
MIME type required by the signed request. |
file_size_bytes |
int |
Maximum allowed size for the object. |
Upload Use-Case Methods
| Method | Description |
|---|---|
create_signed_upload_session |
Creates a new upload session, persists it through UploadSessionRepository, and generates a signed storage URL for the file. |
finalize_upload_session |
Reloads the stored object and validates size, content type, and maximum upload constraints before allowing downstream OCR mapping. |
map_ocr_to_canonical_credit |
Converts OCR-extracted fields into a canonical credit-report structure with provenance and model-adapter payloads. |
build_credit_model_adapter_features |
Aggregates OCR tradelines into the compact adapter feature vector used by the scoring model. |
_extract_tables |
Walks document tables and extracts header/body cells from page objects. |
Signed Upload Flow
create_signed_upload_sessiongeneratesupload_session_id, computesexpires_at, and stores anUploadSession.- The file is uploaded directly to object storage using the returned signed URL.
finalize_upload_sessionfetches the session byupload_session_id,user_id, andbureau.- The blob size and
content_typeare compared against the declared values. map_ocr_to_canonical_creditconverts validated OCR output into a canonical report with:report_metadataconsumerscoressummarymodel_adapterraw_ocr
OCR-to-Canonical Output
map_ocr_to_canonical_credit keeps the uploaded document as traceable evidence and also builds a structured model_adapter payload. The adapter output is what the rest of the pipeline uses for bureau-derived features.
build_credit_model_adapter_features is verified by tests to aggregate tradelines into:
no_of_unsecured_loansvalue_of_unsecured_loansno_of_major_loansvalue_of_major_loansyears_since_first_major_loantradeline_count
and to retain normalized_tradelines entries.
Upload Flow
sequenceDiagram
participant U as Uploader
participant S as signed session use case
participant O as Object storage
participant C as OCR mapper
participant P as scoring bridge
U->>S: create_signed_upload_session
S->>S: persist UploadSession
S-->>U: upload_url and object_path
U->>O: PUT document
U->>S: finalize_upload_session
S->>O: load blob metadata
S-->>U: validated session
U->>C: map_ocr_to_canonical_credit
C-->>P: canonical report and model_adapter
Equifax Instant Intake
File paths: backend/src/usecase/equifax_instant_intake.py, backend/src/interface/transunion_batch_endpoints.py does not apply here, backend/tests/interface/test_equifax_endpoints.py, backend/tests/usecase/test_equifax_instant_intake.py, backend/tests/infra/external_apis/test_equifax_cleaner.py
Equifax intake is the synchronous bureau path. It creates a processing record, fetches identity and credit data from the Equifax client, stores the raw payloads, then returns a normalized summary for scoring.
Request and Result Models
EquifaxInstantIntakeRequest
| Property | Type | Description |
|---|---|---|
phone |
str |
Applicant phone number used for the instant ID lookup. |
date_of_birth |
date |
Date of birth passed to Equifax. |
permissible_purpose |
str |
Consent purpose forwarded to the bureau client. |
EquifaxInstantIntakeRecord
| Property | Type | Description | |
|---|---|---|---|
id |
`int \ | None` | Persistence identifier. |
phone |
str |
Lookup phone number. | |
date_of_birth |
date |
Applicant birth date. | |
status |
str |
Processing state. | |
permissible_purpose |
str |
Consent purpose used for the pull. | |
consent_captured_at |
`datetime \ | None` | Captured consent timestamp. |
instant_id_reference |
`str \ | None` | Equifax instant-ID reference. |
credit_report_reference |
`str \ | None` | Equifax credit-report reference. |
identity_payload |
`dict[str, Any] \ | None` | Raw instant-ID payload. |
credit_report_payload |
`dict[str, Any] \ | None` | Raw credit-report payload. |
error_message |
`str \ | None` | Captured error message for failed pulls. |
created_at |
`datetime \ | None` | Record creation timestamp. |
updated_at |
`datetime \ | None` | Last update timestamp. |
EquifaxInstantIntakeResult
| Property | Type | Description | |
|---|---|---|---|
record |
EquifaxInstantIntakeRecord |
Persisted intake record after status update. | |
identity_profile |
`dict[str, Any] \ | None` | Normalized identity profile. |
credit_report_overview |
`dict[str, Any] \ | None` | Normalized credit-overview fields. |
summary |
dict[str, Any] |
Combined identity and credit summary. | |
sanitized_identity_payload |
dict[str, Any] |
Redacted identity payload for safe storage/return. | |
sanitized_credit_report_payload |
dict[str, Any] |
Redacted credit payload for safe storage/return. | |
data_source |
str |
Source tag returned by the bureau client. |
Equifax Use-Case Methods
| Method | Description |
|---|---|
perform_equifax_instant_intake |
Orchestrates the full Equifax pull, updates status, sanitizes payloads, and returns the normalized result. |
_build_summary |
Merges identity verification state with bureau score and trade-line summary data. |
_build_credit_overview |
Extracts score, score model, reason codes, balances, trade-line counts, and message list from the credit payload. |
Equifax Data Flow
perform_equifax_instant_intake does the following:
- Calls
_ensure_consent_timestampto validate or resolve the consent time. - Creates an
EquifaxInstantIntakeRecordwithstatus="processing". - Calls
equifax_client.fetch_instant_id(...). - Calls
equifax_client.fetch_credit_report(...)using the returned identity payload. - On
EquifaxClientError, marks the recordfailedand records the error message. - Updates the record with request references and full payloads.
- Builds:
identity_profilecredit_overviewsummary
- Returns sanitized copies of both raw payloads for safe downstream use.
Equifax Payload Cleaner
File path: backend/src/infra/external_apis/equifax_cleaner.py
preprocess_equifax_data is the normalization step that removes unsupported fields before the report is analyzed or stored.
Verified behavior from tests:
- Preserves top-level fields such as
CREDIT_AS_OF_DATE. - Whitelists keys under
PPPCL,EADS232, andEADS76. - Normalizes bureau history into
creditHistory. - Maps trade-line details such as:
accountTypedelinquency- payment-history and narrative fields
- Retains
inquiriesas provided.
Equifax API
Perform Equifax Instant Intake
{
"title": "Perform Equifax Instant Intake",
"description": "Creates an intake record, fetches instant identity and credit data from Equifax, persists the raw payloads, and returns a normalized summary.",
"method": "POST",
"baseUrl": "<BackendApiBaseUrl>",
"endpoint": "/api/equifax/instant-intake",
"headers": [
{
"key": "Content-Type",
"value": "application/json",
"required": true
}
],
"queryParams": [],
"pathParams": [],
"bodyType": "json",
"requestBody": "{\n \"phone\": \"+1 (555) 123-4567\",\n \"date_of_birth\": \"1990-04-12\",\n \"consent_captured_at\": \"2024-01-01T00:00:00Z\",\n \"permissible_purpose\": \"credit_application\"\n}",
"formData": [],
"rawBody": "",
"responses": {
"201": {
"description": "Completed intake",
"body": "{\n \"status\": \"completed\",\n \"data_source\": \"equifax\",\n \"identity_profile\": {\n \"requestId\": \"IID-TEST-001\",\n \"matchIndicator\": \"MATCHED\",\n \"confidenceScore\": 0.97,\n \"firstName\": \"Jane\",\n \"lastName\": \"Doe\",\n \"fullName\": \"Jane Q. Doe\",\n \"dateOfBirth\": \"1990-04-12\",\n \"ssn\": {\n \"last4\": \"6789\",\n \"full\": \"123-45-6789\"\n },\n \"addresses\": [\n {\n \"line1\": \"123 Market St\",\n \"city\": \"San Francisco\",\n \"state\": \"CA\",\n \"postalCode\": \"94105\",\n \"country\": \"US\"\n }\n ],\n \"phones\": [\n {\n \"number\": \"+1 (555) 123-4567\",\n \"type\": \"MOBILE\",\n \"lastSeen\": \"2024-03-01\"\n }\n ],\n \"alerts\": [],\n \"notes\": [\n \"Test fixture response\"\n ]\n },\n \"credit_report_overview\": {\n \"model\": \"FICO Score 9\",\n \"score\": 742,\n \"reasonCodes\": [\n \"05\",\n \"10\"\n ],\n \"totalTradeLines\": 12,\n \"openTradeLines\": 9,\n \"revolvingBalance\": 12540,\n \"installmentBalance\": 22100,\n \"derogatoryTradeLines\": 0,\n \"inquiries12Months\": 1,\n \"messages\": [\n \"Test fixture credit report\"\n ]\n },\n \"summary\": {\n \"identityVerified\": true,\n \"confidenceScore\": 0.97,\n \"consumerName\": \"Jane Doe\",\n \"creditScore\": 742,\n \"creditModel\": \"FICO Score 9\",\n \"openTradeLines\": 9,\n \"derogatoryTradeLines\": 0,\n \"inquiries12Months\": 1\n }\n}"
}
}
}
Equifax Intake Flow
sequenceDiagram
participant C as Client
participant A as Equifax API
participant U as perform_equifax_instant_intake
participant R as EquifaxIntakeRepository
C->>U: request payload
U->>R: create processing record
U->>A: fetch_instant_id
A-->>U: identity payload
U->>A: fetch_credit_report
A-->>U: credit payload
U->>R: update_status completed or failed
U-->>C: result with summary and sanitized payloads
TransUnion Batch Processing
File paths: backend/src/usecase/transunion_batch.py, backend/src/interface/transunion_batch_endpoints.py, backend/tests/interface/test_transunion_batch_endpoints.py, backend/tests/infra/importers/test_transunion_batch_parser.py, backend/tests/infra/mysql/test_transunion_batch_repository.py, backend/tests/usecase/test_transunion_batch.py
The TransUnion path is batch-oriented. It accepts a file name, checksum, and a row set, parses each row, persists both clean rows and row-level issues, then returns a summary that the scoring pipeline can ingest.
Batch Record Shapes
TransUnionBatchIngestion
| Property | Type | Description | |
|---|---|---|---|
id |
`int \ | None` | Persistence identifier. |
file_name |
str |
Source batch file name. | |
checksum |
str |
File checksum used for idempotency or audit. | |
status |
str |
Ingestion lifecycle state. | |
total_rows |
int |
Count of input rows. | |
credit_as_of_date |
`date \ | None` | Credit-as-of date attached to the batch. |
metadata |
dict[str, object] |
Arbitrary batch metadata. |
TransUnionBatchRecord
| Property | Type | Description | |
|---|---|---|---|
id |
`int \ | None` | Persistence identifier. |
ingestion_id |
int |
Parent ingestion identifier. | |
perm_id |
str |
Person identifier from the batch row. | |
credit_as_of_date |
`date \ | None` | Row-level credit-as-of date. |
consumer_id |
`str \ | None` | Consumer identifier from the row. |
sequence_number |
`str \ | None` | Sequence value from the row. |
metrics |
dict[str, object] |
Normalized metric dictionary built by the parser. |
TransUnionBatchRowIssue
| Property | Type | Description | |
|---|---|---|---|
id |
`int \ | None` | Persistence identifier. |
ingestion_id |
int |
Parent ingestion identifier. | |
row_number |
int |
One-based row number in the source file. | |
message |
str |
Parsing error message. | |
raw_row |
str |
Original unparsed row text. |
ProcessTransUnionBatchResult
| Property | Type | Description |
|---|---|---|
ingestion |
TransUnionBatchIngestion |
Final ingestion record after processing. |
summary |
dict[str, object] |
Aggregated score summary. |
sanitized_rows |
list[dict[str, object]] |
Masked, API-safe row representations. |
issues |
list[TransUnionBatchRowIssue] |
Row-level parse issues. |
TransUnion Use-Case Methods
| Method | Description |
|---|---|
process_transunion_batch |
Parses rows, stores batch records and issues, computes the summary, and closes the ingestion. |
_build_summary |
Computes average, min, max, and last credit scores from parsed records. |
_build_sanitized_row |
Masks identifiers and returns API-safe row data with metrics. |
Batch Processing Flow
process_transunion_batch runs in a fixed sequence:
- Materializes the input rows into a list.
- Creates a
TransUnionBatchIngestionwithstatus="received". - Updates the ingestion to
status="processing". - For each non-empty row:
- Calls
parse_line. - On parse failure, stores a
TransUnionBatchRowIssue. - On success, creates a
TransUnionBatchRecord.
- Calls
- Persists valid rows and issues.
- Builds
sanitized_rowsusing_build_sanitized_row. - Builds
summaryusing_build_summary. - Marks the ingestion
completedand records counts. - Returns
ProcessTransUnionBatchResult.
Batch Summary Semantics
The summary carries the fields that the broader scoring layer consumes:
dataSource: alwaystransunion_batchtotalRowscreditModel:VantageScore 4.0when vantage scores are presentaverageCreditScoreminCreditScoremaxCreditScorecreditScoretruVisionNewAccountScorewhen available
Row Sanitization
_build_sanitized_row preserves the row-level metrics but masks identifiers:
permIdis maskedconsumerIdis maskedsequenceNumberis passed throughmetricsare copied as dictionaries so API consumers do not receive the original mutable row object
Parser Coverage
backend/tests/infra/importers/test_transunion_batch_parser.py shows that parse_line returns rows with:
perm_idconsumer_idsequence_numbercredit_as_of_datemetrics["vantage_score"]["value"]metrics["vantage_score"]["description"]metrics["truvision_new_account_score"]["value"]
TransUnion API
Create TransUnion Batch Ingestion
{
"title": "Create TransUnion Batch Ingestion",
"description": "Parses TransUnion batch rows, persists the ingestion and row issues, and returns the sanitized batch summary.",
"method": "POST",
"baseUrl": "<BackendApiBaseUrl>",
"endpoint": "/api/transunion/batch-ingestions",
"headers": [
{
"key": "Content-Type",
"value": "application/json",
"required": true
}
],
"queryParams": [],
"pathParams": [],
"bodyType": "json",
"requestBody": "{\n \"file_name\": \"tu_batch.txt\",\n \"checksum\": \"chk-1\",\n \"credit_as_of_date\": \"2023-04-30\",\n \"rows\": [\n \"perm_id|credit_as_of_date|consumer_id|sequence_number|vantage_score|truvision_new_account_score\"\n ],\n \"metadata\": {\n \"triggeredBy\": \"test\"\n }\n}",
"formData": [],
"rawBody": "",
"responses": {
"201": {
"description": "Batch ingestion completed",
"body": "{\n \"status\": \"completed\",\n \"summary\": {\n \"dataSource\": \"transunion_batch\",\n \"totalRows\": 1,\n \"creditModel\": \"VantageScore 4.0\",\n \"averageCreditScore\": 690,\n \"minCreditScore\": 690,\n \"maxCreditScore\": 690,\n \"creditScore\": 690,\n \"truVisionNewAccountScore\": 210\n },\n \"sanitized_rows\": [\n {\n \"creditAsOfDate\": \"2023-04-30\",\n \"permId\": \"******0002\",\n \"consumerId\": \"******0101\",\n \"sequenceNumber\": \"01\",\n \"metrics\": {\n \"vantage_score\": {\n \"value\": 690,\n \"description\": \"VantageScore 4.0\"\n },\n \"truvision_new_account_score\": {\n \"value\": 210,\n \"description\": \"TruVision New Account Score\"\n }\n }\n }\n ],\n \"issues\": []\n}"
}
}
}
Batch Ingestion Flow
sequenceDiagram
participant C as Client
participant E as TransUnion endpoint
participant U as process_transunion_batch
participant P as parse_line
participant R as TransUnionBatchRepository
C->>E: POST batch payload
E->>U: call use case
U->>R: create_ingestion received
U->>R: update_ingestion processing
loop each row
U->>P: parse_line
alt parse failure
U->>R: add_issues
else parse success
U->>R: add_rows
end
end
U->>R: update_ingestion completed
U-->>E: result summary and sanitized rows
E-->>C: 201 response
Japan Staff Credit Pull
File paths: backend/src/usecase/japan_staff_credit.py, backend/tests/usecase/test_japan_staff_credit.py
This use case handles operator-initiated bureau pulls in Japan. The code path is consent-driven and logs the pull outcome for auditability.
Public Methods
| Method | Description |
|---|---|
register_staff_credit_consent |
Registers a staff consent entry for later bureau access. |
perform_staff_credit_pull |
Validates consent and operator authorization, performs the bureau request, and records the pull outcome. |
Observed Behavior
The tests prove the following workflow:
perform_staff_credit_pullrequires a stored consent reference.operator_idmust be authorized for the consent.- The bureau client is called with
permissible_purpose="tenancy_screening". - Successful pulls write a
StaffCreditPullLogwithstatus="succeeded". - The pull result includes the bureau summary, such as
{"score": 712, "grade": "A"}in the fixture.
Error Conditions
The use case imports and surfaces these domain exceptions:
ConsentNotFoundErrorConsentExpiredErrorConsentRevokedErrorBureauRequestErrorOperatorNotAllowedError
Japan Staff Pull Flow
sequenceDiagram
participant O as Operator
participant U as perform_staff_credit_pull
participant CR as StaffCreditConsentRepository
participant LR as StaffCreditPullLogRepository
participant B as Bureau client
O->>U: consent_reference and operator_id
U->>CR: load consent
U->>U: validate consent and operator
U->>B: request_credit_report
B-->>U: BureauCreditReport
U->>LR: store succeeded log
U-->>O: report summary
Canonical Bureau Normalization and Scoring Reconciliation
File paths: backend/src/ml/credit_canonical_schema.py, backend/src/ml/credit_adapters.py, backend/src/ml/credit_variable_map.py, backend/src/ml/credit_model.py, backend/src/ml/credit_pipeline_validation.py, backend/src/usecase/credit_score_runtime.py
This is the bridge between bureau ingestion and the scoring model. The platform does not score raw TransUnion or Equifax payloads directly; it normalizes them into CanonicalBureauData, converts those fields into feature columns, and only then merges them into the final scoring input.
Canonical Bureau Data
CanonicalBureauData
| Property | Type | Description | |
|---|---|---|---|
provider |
str |
Bureau source label such as transunion, equifax, manual_upload, or global. |
|
score_value |
`float \ | None` | Raw score in the source provider scale. |
score_scale_min |
float |
Minimum score scale value. | |
score_scale_max |
float |
Maximum score scale value. | |
score_provider |
str |
Source-specific score model label. | |
utilization_ratio |
`float \ | None` | Overall revolving utilization. |
delinquencies_24m |
int |
Count of delinquency events in the last 24 months. | |
worst_dpd_24m |
int |
Worst days-past-due bucket in the last 24 months. | |
credit_age_months |
`float \ | None` | Age of the newest trade. |
credit_age_months_oldest |
`float \ | None` | Age of the oldest trade. |
inquiries_12m |
int |
Bureau inquiries in the last 12 months. | |
new_accounts_12m |
int |
New accounts in the last 12 months. | |
collections_flag |
bool |
Collection-event flag. | |
collections_count |
int |
Collection count. | |
default_flag |
bool |
Default-event flag. | |
report_recency_days |
float |
Days since the report was pulled. | |
parse_confidence |
float |
Parsing confidence from OCR or ingest. | |
fraud_risk_flag |
bool |
Fraud risk indicator. | |
aml_risk_flag |
bool |
AML risk indicator. | |
estimated_income_score |
`float \ | None` | TransUnion estimated income score. |
revolving_util_patterns |
dict[str, float] |
Trended revolving utilization buckets. | |
bankcard_util_patterns |
dict[str, float] |
Trended bankcard utilization buckets. | |
trade_age_months |
dict[str, float] |
Trade-age buckets by product class. | |
loan_balances |
dict[str, float] |
Loan-balance buckets by product class. | |
raw |
dict[str, Any] |
Original source payload retained for traceability. |
Bureau Adapter Methods
| Method | Description |
|---|---|
BureauAdapter.from_transunion_row |
Normalizes a TransUnion batch row into CanonicalBureauData. |
BureauAdapter.from_equifax_payload |
Normalizes an Equifax instant-intake payload into CanonicalBureauData. |
BureauAdapter.from_manual_report |
Normalizes a manually uploaded or OCR-extracted bureau report into CanonicalBureauData. |
TransUnion Adapter Behavior
BureauAdapter.from_transunion_row accepts both renamed and raw TransUnion field names, and the renamed values win when both are present.
Verified behavior includes:
VantageScoremaps toscore_valueEstimated_Income_Scoremaps toestimated_income_scoreMonths_Since_Oldest_Trade_Openedmaps tocredit_age_months_oldest- Delinquency windows are converted into
worst_dpd_24m revolving_util_patterns,bankcard_util_patterns,trade_age_months, andloan_balancesare populated from both renamed and raw source codesrawpreserves extra fields such asextra_field
Equifax Adapter Behavior
BureauAdapter.from_equifax_payload reads the credit_profile envelope and maps:
risk_score.valuetoscore_valuerisk_score.scale_minandrisk_score.scale_maxto the score rangetradelinesto delinquency counts and worst DPDutilization_ratioorrevolving_utilizationtoutilization_ratioinquiries_12morinquiriestoinquiries_12mcollections_count,default_flag,recency_days,parse_confidence,fraud_risk_flag,aml_risk_flag
Manual Report Adapter Behavior
BureauAdapter.from_manual_report is the path used after OCR extraction. It reads a generic report dictionary with fields such as:
score_valueorscorescore_scale_minscore_scale_maxscore_providerrecency_daysparse_confidence
Model Feature Bridge
The bureau-side fields are reconciled into model features through credit_variable_map.py and credit_model.py. The feature set includes:
bureau_available_flagbureau_score_rawbureau_score_valuebureau_utilizationbureau_utilization_ratiobureau_delinquency_24mbureau_delinquencies_24mbureau_worst_dpd_24mbureau_credit_age_monthsbureau_credit_age_oldest_monthsbureau_inquiries_12mbureau_new_accounts_12mbureau_collections_flagbureau_collections_countbureau_default_flagbureau_defaults_flagbureau_report_recency_daysbureau_parse_confidencebureau_strengthestimated_income_score- trended buckets such as
rev_util_*,bkc_util_*,trade_age_*, andloan_balance_*
credit_score_runtime.py then reads bureau evidence from application.connected_accounts, specifically from:
bureau_reportmanual_credit_reportbureau
and converts that data into runtime features such as:
bureau_available_flagbureau_score_rawbureau_score_valuebureau_report_recency_daysbureau_parse_confidencebureau_strengthbureau_utilizationbureau_delinquency_24mbureau_worst_dpd_24mbureau_credit_age_monthsbureau_credit_age_oldest_monthsbureau_inquiries_12mbureau_new_accounts_12mbureau_collections_flagbureau_collections_countbureau_default_flagbureau_defaults_flagbureau_available_flag
Pipeline Gating
credit_pipeline_validation.py applies deterministic gating before scoring.
Relevant constants:
MIN_DAYS_COVEREDBUREAU_MAX_AGE_DAYSMIN_COMPLETENESS_SCORE_TO_PASS_BUREAU_REQUIRED_PRODUCTS
Classification flags:
- Decline:
KYC_NOT_PASSED,FRAUD_RISK_DETECTED,AML_RISK_DETECTED - Refer:
BUREAU_REQUIRED_BUT_ABSENT,NO_BANKING_DATA,INSUFFICIENT_HISTORY
This means bureau data can be present but still not eligible for scoring if the completeness gate fails or the product requires bureau data and none is available.
Error Handling
The bureau ingestion paths fail differently depending on the source and stage.
| Error or Outcome | Where It Appears | Effect |
|---|---|---|
UploadValidationError |
finalize_upload_session |
Rejects invalid sessions, size mismatches, or content-type mismatches. |
UploadStorageError |
finalize_upload_session |
Rejects missing blobs or missing blob size metadata. |
EquifaxClientError |
perform_equifax_instant_intake |
Marks the intake record failed and captures the error text. |
RuntimeError |
perform_equifax_instant_intake, process_transunion_batch |
Raised when a record is expected to be persisted but is still missing an id. |
| parsing exception | process_transunion_batch |
Converted into a TransUnionBatchRowIssue instead of stopping the whole batch. |
ConsentNotFoundError |
Japan staff pull | Stops the pull when the consent reference is unknown. |
ConsentExpiredError |
Japan staff pull | Stops the pull when the consent is expired. |
ConsentRevokedError |
Japan staff pull | Stops the pull when consent is no longer valid. |
OperatorNotAllowedError |
Japan staff pull | Stops the pull when the operator is not allowed to use the consent. |
BureauRequestError |
Japan staff pull | Captures bureau request failures. |
Testing Considerations
[!NOTE]
process_transunion_batchisolates row-level parse failures intoTransUnionBatchRowIssuerecords and still completes the ingestion for all remaining valid rows.
The supplied tests cover the key bureau-entry behaviors:
test_map_ocr_to_canonical_credit_includes_model_adaptertest_build_credit_model_adapter_features_aggregates_tradelinestest_preprocess_equifax_data_filters_structured_payloadtest_preprocess_equifax_data_handles_tradeline_payloadtest_perform_equifax_instant_intake_generates_summarytest_equifax_instant_intake_endpointtest_process_transunion_batch_creates_summarytest_create_transunion_batch_ingestiontest_parse_line_returns_row_with_core_fieldstest_perform_staff_credit_pull_successtest_perform_staff_credit_pull_rejects_unregistered_operatorTestBureauAdapterTransUnionTestBureauAdapterEquifax
These tests collectively prove that:
- manual upload OCR output is canonicalized before scoring,
- Equifax instant pulls are summarized and sanitized,
- TransUnion batch rows are parsed, persisted, and summarized,
- bureau adapters normalize multiple provider shapes into one schema,
- and the runtime scoring pipeline receives bureau-derived evidence in a format the model can consume.
Dependencies
Core Application Dependencies
FastAPIfor the batch HTTP endpoint layer.SQLAlchemy Sessionfor the TransUnion repository path.UploadSessionRepositoryfor manual-upload session persistence.EquifaxIntakeRepositoryfor instant-intake record lifecycle management.TransUnionBatchRepositoryfor batch ingestion persistence.EquifaxClientfor bureau pulls.FieldSpec,converter_from_data_type, andparse_linefor TransUnion parsing.preprocess_equifax_datafor Equifax payload sanitization.credit_canonical_schema,credit_adapters,credit_model, andcredit_pipeline_validationfor the scoring bridge.
External Services
- Object storage for signed uploads and blob validation.
- Equifax bureau API via
EquifaxClient. - TransUnion batch source files via the importer and repository path.
Logging
backend/src/ml/credit_adapters.py creates a module logger with logging.getLogger(__name__) for adapter-side diagnostics.
Key Classes Reference
| Class | Responsibility |
|---|---|
UploadSession |
Tracks a signed upload request and its storage constraints. |
UploadSessionResponse |
Returns the signed URL and storage metadata for the upload client. |
EquifaxInstantIntakeRecord |
Persists the lifecycle and payloads for an Equifax instant pull. |
EquifaxInstantIntakeRequest |
Captures the request parameters for an Equifax instant intake. |
EquifaxInstantIntakeResult |
Packages the persisted record, normalized profiles, summary, and sanitized payloads. |
TransUnionBatchIngestion |
Represents a batch file lifecycle from receipt to completion. |
TransUnionBatchRecord |
Stores one parsed TransUnion row and its normalized metrics. |
TransUnionBatchRowIssue |
Captures a row-level parse failure without aborting the batch. |
ProcessTransUnionBatchResult |
Returns the completed ingestion, summary, sanitized rows, and issues. |
CanonicalBureauData |
Provides the provider-agnostic bureau schema used by scoring and gating. |
BureauAdapter |
Normalizes TransUnion, Equifax, and manual bureau payloads into canonical data. |