Credit Scoring and Pipeline Domain - Bureau Ingestion, OCR Assisted Upload, and Batch Processing Paths
Overview
This feature area covers the parts of the backend that turn raw bureau evidence into structured credit inputs. It supports three distinct entry paths: TransUnion batch CSV ingestion, Equifax instant-intake normalization, and OCR-assisted manual credit report uploads. Each path converges on a canonical bureau representation or a model-ready feature payload so the broader scoring pipeline can consume the same downstream shape regardless of source.
The batch path stores ingestion state, parses fixed-width bureau rows, captures row-level issues, and produces a sanitised response with bureau metrics. The upload path creates signed storage sessions, validates the uploaded object, maps OCR fields into a canonical credit report, and builds model adapter features from extracted tradelines. The Equifax path preprocesses report data, builds a prompt, and generates an LLM summary from the cleaned report.
Architecture Overview
flowchart TB
subgraph Interface [Interface Layer]
BatchEndpoint[TransUnion batch endpoint]
UploadRequest[Credit report upload request]
CreditReport[Credit report generator]
JapanPull[Japan staff credit pull]
end
subgraph UseCases [Use Case Layer]
ProcessBatch[process_transunion_batch]
CreateSession[create_signed_upload_session]
FinalizeSession[finalize_upload_session]
MapOCR[map_ocr_to_canonical_credit]
GenerateReport[generate_credit_report]
BuildOverview[_build_credit_overview]
BuildSummary[_build_summary]
StaffPull[perform_staff_credit_pull]
end
subgraph Canonical [Canonical Bureau Layer]
BureauData[CanonicalBureauData]
BureauAdapter[BureauAdapter]
end
subgraph Infra [Infrastructure and External Systems]
BatchRepo[SQLTransUnionBatchRepository]
BatchParser[parse_line]
Storage[Cloud Storage]
EquifaxApi[get_credit_report]
Cleaner[preprocess_equifax_data]
Prompt[get_analyze_credit_report_prompt]
LLM[generate_llm_response]
BureauClient[JapanCreditBureauHTTPClient]
end
BatchEndpoint --> ProcessBatch
ProcessBatch --> BatchRepo
ProcessBatch --> BatchParser
ProcessBatch --> BureauData
UploadRequest --> CreateSession
CreateSession --> Storage
FinalizeSession --> Storage
MapOCR --> BureauAdapter
MapOCR --> BureauData
CreditReport --> GenerateReport
GenerateReport --> EquifaxApi
GenerateReport --> Cleaner
GenerateReport --> Prompt
GenerateReport --> LLM
JapanPull --> StaffPull
StaffPull --> BureauClient
BureauAdapter --> BureauData
BuildOverview --> BuildSummary
HTTP API
TransUnion Batch Ingestion
backend/src/interface/transunion_batch_endpoints.py
This endpoint accepts fixed-width TransUnion rows, optionally receives a custom layout description, and forwards the request to process_transunion_batch. It requires a non-empty rows array and returns a completed ingestion result with sanitised rows and row-level issues.
{
"title": "Create TransUnion Batch Ingestion",
"description": "Parses and persists a TransUnion batch file, then returns ingestion status, summary metrics, sanitised rows, and row issues.",
"method": "POST",
"baseUrl": "<BackendApiBaseUrl>",
"endpoint": "/api/transunion/batch-ingestions",
"headers": [
{
"key": "Content-Type",
"value": "application/json",
"required": true
}
],
"queryParams": [],
"pathParams": [],
"bodyType": "json",
"requestBody": "{\n \"file_name\": \"tu_batch_2023_04_30.txt\",\n \"checksum\": \"chk-20230430-acde1234\",\n \"credit_as_of_date\": \"2023-04-30\",\n \"rows\": [\n \"PERM0002 2023-04-30 101 01 690 210 \"\n ],\n \"metadata\": {\n \"triggeredBy\": \"manual_upload\",\n \"source\": \"batch_job\"\n },\n \"layout\": [\n {\n \"key\": \"PPPCL\",\n \"start\": 1,\n \"end\": 12,\n \"normalised_key\": \"perm_id\",\n \"description\": \"Scrambled ID\",\n \"data_group\": \"PPPCL\",\n \"field_name\": \"PermID\",\n \"length\": 12,\n \"data_type\": \"string\"\n }\n ]\n}",
"formData": [],
"rawBody": "",
"responses": {
"201": {
"description": "Successful ingestion",
"body": "{\n \"ingestion_id\": 42,\n \"status\": \"completed\",\n \"total_rows\": 1,\n \"processed_rows\": 1,\n \"error_count\": 0,\n \"credit_as_of_date\": \"2023-04-30\",\n \"summary\": {\n \"dataSource\": \"transunion_batch\",\n \"totalRows\": 1,\n \"creditModel\": \"VantageScore 4.0\",\n \"averageCreditScore\": 690,\n \"minCreditScore\": 690,\n \"maxCreditScore\": 690,\n \"creditScore\": 690,\n \"truVisionNewAccountScore\": 210\n },\n \"sanitized_rows\": [\n {\n \"creditAsOfDate\": \"2023-04-30\",\n \"permId\": \"********0002\",\n \"consumerId\": \"********0101\",\n \"sequenceNumber\": \"01\",\n \"metrics\": {\n \"vantage_score\": {\n \"value\": 690,\n \"description\": \"VantageScore 4.0\"\n },\n \"truvision_new_account_score\": {\n \"value\": 210\n }\n }\n }\n ],\n \"issues\": []\n}"
},
"400": {
"description": "Rejected when rows are missing",
"body": "{\n \"detail\": \"At least one row is required\"\n}"
}
}
}
Bureau Normalization and Canonical Data
CanonicalBureauData
backend/src/ml/credit_canonical_schema.py
CanonicalBureauData is the shared bureau shape used by TransUnion batch rows, Equifax instant-intake payloads, and manually uploaded OCR reports. The adapter layer is responsible for populating the provider-specific fields before the scoring pipeline consumes them.
| Property | Type | Description |
|---|---|---|
provider |
str |
Bureau source label such as transunion, equifax, manual_upload, or global. |
score_value |
`float \ | None` |
score_scale_min |
float |
Lower bound of the provider score scale. |
score_scale_max |
float |
Upper bound of the provider score scale. |
score_provider |
str |
Provider-specific score label such as vantage_4 or equifax_risk. |
utilization_ratio |
`float \ | None` |
delinquencies_24m |
int |
Number of delinquency events in the past 24 months. |
worst_dpd_24m |
int |
Worst days-past-due bucket in the past 24 months. |
credit_age_months |
`float \ | None` |
credit_age_months_oldest |
`float \ | None` |
inquiries_12m |
int |
Inquiry count in the past 12 months. |
new_accounts_12m |
int |
New account count in the past 12 months. |
collections_flag |
bool |
Indicates whether collections are present. |
collections_count |
int |
Count of collections. |
default_flag |
bool |
Indicates whether a default is present. |
report_recency_days |
float |
Age of the report in days. |
parse_confidence |
float |
Parser or OCR confidence from 0 to 1. |
fraud_risk_flag |
bool |
Fraud-risk indicator. |
aml_risk_flag |
bool |
AML-risk indicator. |
estimated_income_score |
`float \ | None` |
revolving_util_patterns |
dict[str, float] |
TransUnion trended revolving utilisation patterns. |
bankcard_util_patterns |
dict[str, float] |
TransUnion trended bankcard utilisation patterns. |
trade_age_months |
dict[str, float] |
TransUnion trade-age breakdown. |
loan_balances |
dict[str, float] |
TransUnion loan balance breakdown. |
raw |
dict[str, Any] |
Original provider payload or row retained for traceability. |
BureauAdapter
backend/src/ml/credit_adapters.py
BureauAdapter converts provider-specific bureau payloads into CanonicalBureauData.
| Method | Description |
|---|---|
from_transunion_row |
Normalises a TransUnion batch row, accepting both raw field codes and renamed columns. |
from_equifax_payload |
Normalises an Equifax instant-intake response into canonical bureau data. |
from_manual_report |
Normalises OCR-extracted or manually uploaded bureau report data. |
Key normalisation rules visible in code:
from_transunion_rowprefers renamed TransUnion fields such asVantageScore,Estimated_Income_Score, andMonths_Since_Oldest_Trade_Opened, but falls back to raw codes likeevtg04_finscore,etie04_score, andeads142_at20s.from_equifax_payloadreadscredit_profile.risk_score,credit_profile.tradelines,utilization_ratioorrevolving_utilization, anddays_past_duevalues to derive delinquency and age metrics.from_manual_reportis the OCR/manual-upload path used by the credit upload flow and preserves parser confidence, utilisation, inquiries, and default indicators.
TransUnion Batch Processing
process_transunion_batch
backend/src/usecase/transunion_batch.py
| Method | Description |
|---|---|
process_transunion_batch |
Parses batch rows, persists the ingestion lifecycle, records row issues, and returns the completed summary and sanitised rows. |
The processing flow is:
- Creates a
TransUnionBatchIngestionwithstatus="received". - Requires the persisted ingestion to have an
id; otherwise it raisesRuntimeError("Ingestion must be persisted"). - Updates the ingestion to
status="processing"before row parsing begins. - Iterates rows with 1-based line numbers and skips blank lines.
- Calls
parse_linefor each non-empty row. - Converts parse exceptions into
TransUnionBatchRowIssueobjects instead of aborting the whole file. - Persists valid rows with
repository.add_rows(...)and issues withrepository.add_issues(...). - Builds sanitised rows with masked identifiers and a summary over processed records.
- Updates the ingestion to
status="completed"and writes aggregate counts and summary fields back to storage.
Internal summary and row sanitisation
_build_summary calculates:
dataSource: "transunion_batch"totalRowscreditModel: "VantageScore 4.0"when Vantage values are presentaverageCreditScore,minCreditScore,maxCreditScorecreditScorefrom the last parsed Vantage scoretruVisionNewAccountScorefrom the last parsed TruVision score
_build_sanitized_row returns a public-safe row shape with:
creditAsOfDate- masked
permId - masked
consumerId sequenceNumbermetricscopied from the parsed row
ProcessTransUnionBatchResult
backend/src/usecase/transunion_batch.py
| Property | Type | Description |
|---|---|---|
ingestion |
TransUnionBatchIngestion |
Updated ingestion record returned from the repository. |
summary |
Dict[str, object] |
Aggregate metrics derived from parsed rows. |
sanitized_rows |
List[Dict[str, object]] |
Public-safe row payloads with masked identifiers. |
issues |
List[TransUnionBatchRowIssue] |
Row-level parse failures captured during processing. |
TransUnionBatchIngestionRequestSchema
backend/src/interface/schemas.py
| Property | Type | Description |
|---|---|---|
file_name |
str |
Input file name. |
checksum |
str |
File checksum for deduplication or traceability. |
credit_as_of_date |
Optional[date] |
Optional batch-level report date. |
rows |
List[str] |
Fixed-width input rows. |
metadata |
Dict[str, Any] |
Arbitrary metadata persisted with the ingestion. |
layout |
Optional[List[TransUnionLayoutFieldSchema]] |
Optional custom field layout for parsing. |
TransUnionBatchIngestionResponseSchema
backend/src/interface/schemas.py
| Property | Type | Description |
|---|---|---|
ingestion_id |
int |
Persisted ingestion identifier. |
status |
str |
Lifecycle status after processing. |
total_rows |
int |
Total counted rows for the ingestion. |
processed_rows |
int |
Number of successfully parsed rows. |
error_count |
int |
Number of parse failures. |
credit_as_of_date |
Optional[date] |
Batch-level report date returned from storage. |
summary |
Dict[str, Any] |
Aggregated scoring summary. |
sanitized_rows |
List[TransUnionBatchSanitizedRowSchema] |
Public-safe parsed rows. |
issues |
List[TransUnionBatchIssueSchema] |
Row issues with line number and message. |
TransUnion batch parser behavior
backend/tests/infra/importers/test_transunion_batch_parser.py
The tests show the parser contract used by the ingestion path:
parse_linereturns parsed identifiers,credit_as_of_date, and per-metric values.parse_lineraisesValueErrorwhen date values are invalid.load_layout_from_csvmaps CSV layout rows intoFieldSpecinstances withnormalised_keyvalues.- The parser supports both default layout parsing and caller-provided layout parsing.
OCR Assisted Upload
CreditReportUploadCompleteRequest
backend/src/interface/schemas.py
This request object captures the final step after a user uploads a manual credit report and confirms extracted OCR fields.
| Property | Type | Description |
|---|---|---|
user_id |
str |
User who owns the upload session. |
bureau |
str |
Bureau label for the uploaded report. |
ocr_payload |
Dict[str, Any> |
OCR output used to build the canonical credit report. |
user_confirmed |
bool |
User confirmation flag for the extracted fields. |
confirmed_at |
Optional[datetime] |
Optional confirmation timestamp. |
create_signed_upload_session
backend/src/usecase/credit_upload.py
| Method | Description |
|---|---|
create_signed_upload_session |
Creates an upload session record, stores it in the repository, and returns a signed PUT URL for object storage. |
finalize_upload_session
backend/src/usecase/credit_upload.py
| Method | Description |
|---|---|
finalize_upload_session |
Validates that the uploaded object exists, matches the declared size and content type, and returns the active upload session. |
map_ocr_to_canonical_credit
backend/src/usecase/credit_upload.py
| Method | Description |
|---|---|
map_ocr_to_canonical_credit |
Converts OCR payload fields into a canonical credit report with metadata, consumer data, score fields, summary fields, model adapter features, and raw OCR payload. |
build_credit_model_adapter_features
backend/src/usecase/credit_upload.py
| Method | Description |
|---|---|
build_credit_model_adapter_features |
Aggregates OCR tradelines into a compact adapter payload and model feature vector used by downstream credit scoring. |
OCR field and tradeline extraction
The OCR upload path uses internal helpers to prepare the canonical report:
_index_ocr_fieldsindexesocr_payload["fields"]by a normalised key and also adds top-level scalar values._extract_tradelinesacceptstradelines,trade_lines,accounts, andloan_accounts, then keeps only dictionary-like entries._extract_tablesconverts extracted page tables into{headers, rows}records for downstream use.map_ocr_to_canonical_creditresolves field aliases such asfull_nameorname,date_of_birthordob,ssnorsocial_security_number, andscoreorcredit_scoreorfico_score.
The returned canonical report contains:
report_metadatawith session, bureau, user, country, object path, ingest timestamp, and user confirmation detailsconsumerfieldsscoreslistsummaryfieldsmodel_adapterraw_ocr
Equifax Credit Report Generation and Instant Intake
generate_credit_report
backend/src/usecase/credit_report.py
| Method | Description |
|---|---|
generate_credit_report |
Logs in to the external credit service, fetches the report, preprocesses it, builds the analysis prompt, and returns the LLM summary plus the raw report payload. |
Equifax instant intake summary helpers
backend/src/usecase/equifax_instant_intake.py
_build_credit_overviewextracts:modelscorereasonCodestotalTradeLinesopenTradeLinesrevolvingBalanceinstallmentBalancederogatoryTradeLinesinquiries12Monthsmessages
_build_summarycombines identity and credit overview data into:identityVerifiedconfidenceScoreconsumerNamecreditScorecreditModelopenTradeLinesderogatoryTradeLinesinquiries12Months
equifax_cleaner and prompt integration
[!NOTE]
generate_credit_reportreadsrequest_idfromreport_dataand importsget_pdf_report, but the active return path only emits{"llm_summary": llm_summary, "raw_report": report_data}. The PDF fetch line is commented out, sopdf_contentnever contributes to the output.
The Equifax report path calls preprocess_equifax_data before the prompt is built. The cleaned payload is then passed to get_analyze_credit_report_prompt and finally to generate_llm_response, so the LLM summary always runs on sanitised report data rather than the raw upstream payload.
Japan Staff Credit Pull
backend/src/usecase/japan_staff_credit.py
backend/src/interface/japan_credit_endpoints.py
This path handles staff-initiated bureau pulls for Japan. The endpoint constructs StaffCreditPullRequest with consent_reference, operator_id, requested_at, optional device_fingerprint, and optional permissible_purpose, then calls perform_staff_credit_pull.
Visible error mapping in the endpoint:
ConsentNotFoundError->404ConsentExpiredError,ConsentRevokedError,OperatorNotAllowedError->400- Any other exception ->
502
The successful response includes:
consent_referenceoperator_idrequested_atbureau_referencesummarydata_sourcecreated_atreport
Feature Flows
TransUnion batch ingestion flow
sequenceDiagram
participant C as Client
participant E as TransUnion Batch Endpoint
participant U as process_transunion_batch
participant R as SQLTransUnionBatchRepository
participant P as parse_line
C->>E: POST batch ingestions
E->>E: validate rows present
E->>U: process_transunion_batch
U->>R: create_ingestion received
U->>R: update_ingestion processing
loop each non-empty row
U->>P: parse_line
alt parse error
U->>U: append issue
else parsed row
U->>U: collect record
end
end
U->>R: add_rows
U->>R: add_issues
U->>R: update_ingestion completed
R-->>U: updated ingestion
U-->>E: result
E-->>C: 201 response
OCR assisted upload and canonical mapping flow
sequenceDiagram
participant C as Client
participant U as create_signed_upload_session
participant S as Cloud Storage
participant F as finalize_upload_session
participant M as map_ocr_to_canonical_credit
C->>U: request upload session
U->>S: generate signed PUT URL
S-->>U: upload session response
U-->>C: signed URL and object path
C->>S: upload report file
C->>F: finalize session
F->>S: validate object size and content type
S-->>F: object metadata
F-->>C: active upload session
C->>M: submit OCR payload
M->>M: index OCR fields
M->>M: map consumer and score fields
M->>M: build model adapter features
M-->>C: canonical credit report
Equifax credit report generation flow
sequenceDiagram
participant C as Client
participant G as generate_credit_report
participant A as login_user
participant B as get_credit_report
participant N as preprocess_equifax_data
participant P as get_analyze_credit_report_prompt
participant L as generate_llm_response
C->>G: submit user data
G->>A: login_user
A-->>G: token
G->>B: get_credit_report
B-->>G: report data
G->>N: preprocess_equifax_data
N-->>G: cleaned report
G->>P: get_analyze_credit_report_prompt
P-->>G: prompt
G->>L: generate_llm_response
L-->>G: summary
G-->>C: llm_summary and raw_report
Japan staff credit pull flow
sequenceDiagram
participant C as Client
participant E as initiate_credit_pull
participant U as perform_staff_credit_pull
participant CR as consent repository
participant LR as log repository
participant B as JapanCreditBureauHTTPClient
C->>E: submit consent reference and operator data
E->>U: perform_staff_credit_pull
U->>CR: verify consent
U->>LR: write pull log
U->>B: request bureau report
B-->>U: bureau report
U-->>E: result bundle
E-->>C: pull response
State Management
TransUnion ingestion lifecycle
The TransUnion batch path stores and updates an explicit ingestion status:
receivedis written during creationprocessingis written before row parsing beginscompletedis written after rows and issues are persisted
OCR upload session lifecycle
The upload path uses session metadata to keep the manual upload flow consistent:
upload_session_iduser_idbureaucountryobject_pathcontent_typefile_size_bytesexpires_at
The canonical OCR report also stores confirmation state under report_metadata.user_confirmation with:
confirmedconfirmed_at
Equifax and Japan pull state
The Equifax report generator is a one-shot transform from fetched report data to summary output. The Japan staff pull path carries consent and pull-log state through repository calls and bureau-request logging.
Error Handling
TransUnion batch ingestion
- Empty
rowsarray returns HTTP400withAt least one row is required. - Parse failures are converted into
TransUnionBatchRowIssueentries instead of aborting the batch. - A missing persisted ingestion id raises
RuntimeError("Ingestion must be persisted").
OCR upload
_get_bucket_nameraisesUploadConfigErrorwhenCREDIT_REPORT_UPLOAD_BUCKETis not set.finalize_upload_sessionraisesUploadValidationErrorwhen the upload session cannot be found or the object does not match expected constraints.finalize_upload_sessionraisesUploadStorageErrorwhen the object is missing or its size cannot be read.- Validation failures delete the uploaded blob before raising.
Equifax generation
- The active report generation path does not catch downstream login, fetch, preprocessing, or LLM failures in this file; they propagate to the caller.
Japan staff credit pull
ConsentNotFoundErrormaps to404.ConsentExpiredError,ConsentRevokedError, andOperatorNotAllowedErrormap to400.- Unexpected integration failures map to
502.
Dependencies
Domain and repository dependencies
TransUnionBatchIngestionTransUnionBatchRecordTransUnionBatchRowIssueTransUnionBatchRepositorySQLTransUnionBatchRepositoryUploadSessionRepositoryHousingApplicationis not part of this section and is not used here
Parser and transform dependencies
parse_lineload_layout_from_csvFieldSpecconverter_from_data_type
External service dependencies
get_credit_reportlogin_userget_pdf_reportpreprocess_equifax_datagenerate_llm_responseget_analyze_credit_report_promptJapanCreditBureauHTTPClient- Cloud Storage client used by
create_signed_upload_sessionandfinalize_upload_session
Pipeline consumers
CanonicalBureauDatabuild_credit_model_adapter_features- downstream credit scoring and contract layers that consume canonical bureau evidence
Testing Considerations
test_create_transunion_batch_ingestionverifies the batch endpoint returns201, a completed status, and masked row output.test_create_transunion_batch_ingestion_with_custom_layoutverifies custom layout conversion and parsing.test_process_transunion_batch_creates_summaryverifies TransUnion summary generation, includingcreditModel,creditScore, and sanitised identifiers.test_parse_line_returns_row_with_core_fieldsverifies parser field extraction and metric values.test_parse_line_raises_for_invalid_dateverifies invalid dates raiseValueError.test_load_layout_from_csv_maps_core_fields_without_manual_normalised_keysverifies CSV layout loading and layout-driven parsing.test_map_ocr_to_canonical_credit_includes_model_adapterverifies OCR mapping and model adapter generation.test_build_credit_model_adapter_features_aggregates_tradelinesverifies tradeline aggregation into adapter features.test_preprocess_equifax_data_handles_tradeline_payloadandtest_preprocess_equifax_data_filters_structured_payloadverify the Equifax cleaning step that feedsgenerate_credit_report.test_perform_staff_credit_pull_successverifies the Japan staff pull path succeeds and logs the request.
Key Classes Reference
| Class | Responsibility |
|---|---|
transunion_batch_endpoints.py |
Accepts TransUnion batch ingestion requests and returns processed ingestion results. |
transunion_batch.py |
Parses and persists TransUnion batch ingestions and builds summary output. |
credit_canonical_schema.py |
Defines the canonical bureau data shape shared by all bureau paths. |
credit_adapters.py |
Normalises TransUnion, Equifax, and manual bureau payloads into canonical bureau data. |
credit_upload.py |
Manages signed upload sessions and converts OCR payloads into canonical credit data. |
credit_report.py |
Orchestrates Equifax-style report retrieval, cleaning, prompting, and summary generation. |
equifax_instant_intake.py |
Builds Equifax credit overview and summary payloads from instant-intake responses. |
japan_staff_credit.py |
Drives Japan staff-initiated bureau pull orchestration. |
schemas.py |
Defines request and response schemas for batch ingestion and OCR completion flows. |
test_transunion_batch_parser.py |
Verifies batch parser field extraction, layout loading, and invalid date handling. |