Credit Scoring and Pipeline Domain - Bureau Ingestion, OCR Assisted Upload, and Batch Processing Paths

Overview

This feature area covers the parts of the backend that turn raw bureau evidence into structured credit inputs. It supports three distinct entry paths: TransUnion batch CSV ingestion, Equifax instant-intake normalization, and OCR-assisted manual credit report uploads. Each path converges on a canonical bureau representation or a model-ready feature payload so the broader scoring pipeline can consume the same downstream shape regardless of source.

The batch path stores ingestion state, parses fixed-width bureau rows, captures row-level issues, and produces a sanitised response with bureau metrics. The upload path creates signed storage sessions, validates the uploaded object, maps OCR fields into a canonical credit report, and builds model adapter features from extracted tradelines. The Equifax path preprocesses report data, builds a prompt, and generates an LLM summary from the cleaned report.

Architecture Overview

flowchart TB
    subgraph Interface [Interface Layer]
        BatchEndpoint[TransUnion batch endpoint]
        UploadRequest[Credit report upload request]
        CreditReport[Credit report generator]
        JapanPull[Japan staff credit pull]
    end

    subgraph UseCases [Use Case Layer]
        ProcessBatch[process_transunion_batch]
        CreateSession[create_signed_upload_session]
        FinalizeSession[finalize_upload_session]
        MapOCR[map_ocr_to_canonical_credit]
        GenerateReport[generate_credit_report]
        BuildOverview[_build_credit_overview]
        BuildSummary[_build_summary]
        StaffPull[perform_staff_credit_pull]
    end

    subgraph Canonical [Canonical Bureau Layer]
        BureauData[CanonicalBureauData]
        BureauAdapter[BureauAdapter]
    end

    subgraph Infra [Infrastructure and External Systems]
        BatchRepo[SQLTransUnionBatchRepository]
        BatchParser[parse_line]
        Storage[Cloud Storage]
        EquifaxApi[get_credit_report]
        Cleaner[preprocess_equifax_data]
        Prompt[get_analyze_credit_report_prompt]
        LLM[generate_llm_response]
        BureauClient[JapanCreditBureauHTTPClient]
    end

    BatchEndpoint --> ProcessBatch
    ProcessBatch --> BatchRepo
    ProcessBatch --> BatchParser
    ProcessBatch --> BureauData

    UploadRequest --> CreateSession
    CreateSession --> Storage
    FinalizeSession --> Storage
    MapOCR --> BureauAdapter
    MapOCR --> BureauData

    CreditReport --> GenerateReport
    GenerateReport --> EquifaxApi
    GenerateReport --> Cleaner
    GenerateReport --> Prompt
    GenerateReport --> LLM

    JapanPull --> StaffPull
    StaffPull --> BureauClient

    BureauAdapter --> BureauData
    BuildOverview --> BuildSummary

HTTP API

TransUnion Batch Ingestion

backend/src/interface/transunion_batch_endpoints.py

This endpoint accepts fixed-width TransUnion rows, optionally receives a custom layout description, and forwards the request to process_transunion_batch. It requires a non-empty rows array and returns a completed ingestion result with sanitised rows and row-level issues.

{
    "title": "Create TransUnion Batch Ingestion",
    "description": "Parses and persists a TransUnion batch file, then returns ingestion status, summary metrics, sanitised rows, and row issues.",
    "method": "POST",
    "baseUrl": "<BackendApiBaseUrl>",
    "endpoint": "/api/transunion/batch-ingestions",
    "headers": [
        {
            "key": "Content-Type",
            "value": "application/json",
            "required": true
        }
    ],
    "queryParams": [],
    "pathParams": [],
    "bodyType": "json",
    "requestBody": "{\n    \"file_name\": \"tu_batch_2023_04_30.txt\",\n    \"checksum\": \"chk-20230430-acde1234\",\n    \"credit_as_of_date\": \"2023-04-30\",\n    \"rows\": [\n        \"PERM0002     2023-04-30 101       01 690 210                                                 \"\n    ],\n    \"metadata\": {\n        \"triggeredBy\": \"manual_upload\",\n        \"source\": \"batch_job\"\n    },\n    \"layout\": [\n        {\n            \"key\": \"PPPCL\",\n            \"start\": 1,\n            \"end\": 12,\n            \"normalised_key\": \"perm_id\",\n            \"description\": \"Scrambled ID\",\n            \"data_group\": \"PPPCL\",\n            \"field_name\": \"PermID\",\n            \"length\": 12,\n            \"data_type\": \"string\"\n        }\n    ]\n}",
    "formData": [],
    "rawBody": "",
    "responses": {
        "201": {
            "description": "Successful ingestion",
            "body": "{\n    \"ingestion_id\": 42,\n    \"status\": \"completed\",\n    \"total_rows\": 1,\n    \"processed_rows\": 1,\n    \"error_count\": 0,\n    \"credit_as_of_date\": \"2023-04-30\",\n    \"summary\": {\n        \"dataSource\": \"transunion_batch\",\n        \"totalRows\": 1,\n        \"creditModel\": \"VantageScore 4.0\",\n        \"averageCreditScore\": 690,\n        \"minCreditScore\": 690,\n        \"maxCreditScore\": 690,\n        \"creditScore\": 690,\n        \"truVisionNewAccountScore\": 210\n    },\n    \"sanitized_rows\": [\n        {\n            \"creditAsOfDate\": \"2023-04-30\",\n            \"permId\": \"********0002\",\n            \"consumerId\": \"********0101\",\n            \"sequenceNumber\": \"01\",\n            \"metrics\": {\n                \"vantage_score\": {\n                    \"value\": 690,\n                    \"description\": \"VantageScore 4.0\"\n                },\n                \"truvision_new_account_score\": {\n                    \"value\": 210\n                }\n            }\n        }\n    ],\n    \"issues\": []\n}"
        },
        "400": {
            "description": "Rejected when rows are missing",
            "body": "{\n    \"detail\": \"At least one row is required\"\n}"
        }
    }
}

Bureau Normalization and Canonical Data

CanonicalBureauData

backend/src/ml/credit_canonical_schema.py

CanonicalBureauData is the shared bureau shape used by TransUnion batch rows, Equifax instant-intake payloads, and manually uploaded OCR reports. The adapter layer is responsible for populating the provider-specific fields before the scoring pipeline consumes them.

Property Type Description
provider str Bureau source label such as transunion, equifax, manual_upload, or global.
score_value `float \ None`
score_scale_min float Lower bound of the provider score scale.
score_scale_max float Upper bound of the provider score scale.
score_provider str Provider-specific score label such as vantage_4 or equifax_risk.
utilization_ratio `float \ None`
delinquencies_24m int Number of delinquency events in the past 24 months.
worst_dpd_24m int Worst days-past-due bucket in the past 24 months.
credit_age_months `float \ None`
credit_age_months_oldest `float \ None`
inquiries_12m int Inquiry count in the past 12 months.
new_accounts_12m int New account count in the past 12 months.
collections_flag bool Indicates whether collections are present.
collections_count int Count of collections.
default_flag bool Indicates whether a default is present.
report_recency_days float Age of the report in days.
parse_confidence float Parser or OCR confidence from 0 to 1.
fraud_risk_flag bool Fraud-risk indicator.
aml_risk_flag bool AML-risk indicator.
estimated_income_score `float \ None`
revolving_util_patterns dict[str, float] TransUnion trended revolving utilisation patterns.
bankcard_util_patterns dict[str, float] TransUnion trended bankcard utilisation patterns.
trade_age_months dict[str, float] TransUnion trade-age breakdown.
loan_balances dict[str, float] TransUnion loan balance breakdown.
raw dict[str, Any] Original provider payload or row retained for traceability.

BureauAdapter

backend/src/ml/credit_adapters.py

BureauAdapter converts provider-specific bureau payloads into CanonicalBureauData.

Method Description
from_transunion_row Normalises a TransUnion batch row, accepting both raw field codes and renamed columns.
from_equifax_payload Normalises an Equifax instant-intake response into canonical bureau data.
from_manual_report Normalises OCR-extracted or manually uploaded bureau report data.

Key normalisation rules visible in code:

  • from_transunion_row prefers renamed TransUnion fields such as VantageScore, Estimated_Income_Score, and Months_Since_Oldest_Trade_Opened, but falls back to raw codes like evtg04_finscore, etie04_score, and eads142_at20s.
  • from_equifax_payload reads credit_profile.risk_score, credit_profile.tradelines, utilization_ratio or revolving_utilization, and days_past_due values to derive delinquency and age metrics.
  • from_manual_report is the OCR/manual-upload path used by the credit upload flow and preserves parser confidence, utilisation, inquiries, and default indicators.

TransUnion Batch Processing

process_transunion_batch

backend/src/usecase/transunion_batch.py

Method Description
process_transunion_batch Parses batch rows, persists the ingestion lifecycle, records row issues, and returns the completed summary and sanitised rows.

The processing flow is:

  • Creates a TransUnionBatchIngestion with status="received".
  • Requires the persisted ingestion to have an id; otherwise it raises RuntimeError("Ingestion must be persisted").
  • Updates the ingestion to status="processing" before row parsing begins.
  • Iterates rows with 1-based line numbers and skips blank lines.
  • Calls parse_line for each non-empty row.
  • Converts parse exceptions into TransUnionBatchRowIssue objects instead of aborting the whole file.
  • Persists valid rows with repository.add_rows(...) and issues with repository.add_issues(...).
  • Builds sanitised rows with masked identifiers and a summary over processed records.
  • Updates the ingestion to status="completed" and writes aggregate counts and summary fields back to storage.

Internal summary and row sanitisation

_build_summary calculates:

  • dataSource: "transunion_batch"
  • totalRows
  • creditModel: "VantageScore 4.0" when Vantage values are present
  • averageCreditScore, minCreditScore, maxCreditScore
  • creditScore from the last parsed Vantage score
  • truVisionNewAccountScore from the last parsed TruVision score

_build_sanitized_row returns a public-safe row shape with:

  • creditAsOfDate
  • masked permId
  • masked consumerId
  • sequenceNumber
  • metrics copied from the parsed row

ProcessTransUnionBatchResult

backend/src/usecase/transunion_batch.py

Property Type Description
ingestion TransUnionBatchIngestion Updated ingestion record returned from the repository.
summary Dict[str, object] Aggregate metrics derived from parsed rows.
sanitized_rows List[Dict[str, object]] Public-safe row payloads with masked identifiers.
issues List[TransUnionBatchRowIssue] Row-level parse failures captured during processing.

TransUnionBatchIngestionRequestSchema

backend/src/interface/schemas.py

Property Type Description
file_name str Input file name.
checksum str File checksum for deduplication or traceability.
credit_as_of_date Optional[date] Optional batch-level report date.
rows List[str] Fixed-width input rows.
metadata Dict[str, Any] Arbitrary metadata persisted with the ingestion.
layout Optional[List[TransUnionLayoutFieldSchema]] Optional custom field layout for parsing.

TransUnionBatchIngestionResponseSchema

backend/src/interface/schemas.py

Property Type Description
ingestion_id int Persisted ingestion identifier.
status str Lifecycle status after processing.
total_rows int Total counted rows for the ingestion.
processed_rows int Number of successfully parsed rows.
error_count int Number of parse failures.
credit_as_of_date Optional[date] Batch-level report date returned from storage.
summary Dict[str, Any] Aggregated scoring summary.
sanitized_rows List[TransUnionBatchSanitizedRowSchema] Public-safe parsed rows.
issues List[TransUnionBatchIssueSchema] Row issues with line number and message.

TransUnion batch parser behavior

backend/tests/infra/importers/test_transunion_batch_parser.py

The tests show the parser contract used by the ingestion path:

  • parse_line returns parsed identifiers, credit_as_of_date, and per-metric values.
  • parse_line raises ValueError when date values are invalid.
  • load_layout_from_csv maps CSV layout rows into FieldSpec instances with normalised_key values.
  • The parser supports both default layout parsing and caller-provided layout parsing.

OCR Assisted Upload

CreditReportUploadCompleteRequest

backend/src/interface/schemas.py

This request object captures the final step after a user uploads a manual credit report and confirms extracted OCR fields.

Property Type Description
user_id str User who owns the upload session.
bureau str Bureau label for the uploaded report.
ocr_payload Dict[str, Any> OCR output used to build the canonical credit report.
user_confirmed bool User confirmation flag for the extracted fields.
confirmed_at Optional[datetime] Optional confirmation timestamp.

create_signed_upload_session

backend/src/usecase/credit_upload.py

Method Description
create_signed_upload_session Creates an upload session record, stores it in the repository, and returns a signed PUT URL for object storage.

finalize_upload_session

backend/src/usecase/credit_upload.py

Method Description
finalize_upload_session Validates that the uploaded object exists, matches the declared size and content type, and returns the active upload session.

map_ocr_to_canonical_credit

backend/src/usecase/credit_upload.py

Method Description
map_ocr_to_canonical_credit Converts OCR payload fields into a canonical credit report with metadata, consumer data, score fields, summary fields, model adapter features, and raw OCR payload.

build_credit_model_adapter_features

backend/src/usecase/credit_upload.py

Method Description
build_credit_model_adapter_features Aggregates OCR tradelines into a compact adapter payload and model feature vector used by downstream credit scoring.

OCR field and tradeline extraction

The OCR upload path uses internal helpers to prepare the canonical report:

  • _index_ocr_fields indexes ocr_payload["fields"] by a normalised key and also adds top-level scalar values.
  • _extract_tradelines accepts tradelines, trade_lines, accounts, and loan_accounts, then keeps only dictionary-like entries.
  • _extract_tables converts extracted page tables into {headers, rows} records for downstream use.
  • map_ocr_to_canonical_credit resolves field aliases such as full_name or name, date_of_birth or dob, ssn or social_security_number, and score or credit_score or fico_score.

The returned canonical report contains:

  • report_metadata with session, bureau, user, country, object path, ingest timestamp, and user confirmation details
  • consumer fields
  • scores list
  • summary fields
  • model_adapter
  • raw_ocr

Equifax Credit Report Generation and Instant Intake

generate_credit_report

backend/src/usecase/credit_report.py

Method Description
generate_credit_report Logs in to the external credit service, fetches the report, preprocesses it, builds the analysis prompt, and returns the LLM summary plus the raw report payload.

Equifax instant intake summary helpers

backend/src/usecase/equifax_instant_intake.py

  • _build_credit_overview extracts:
    • model
    • score
    • reasonCodes
    • totalTradeLines
    • openTradeLines
    • revolvingBalance
    • installmentBalance
    • derogatoryTradeLines
    • inquiries12Months
    • messages
  • _build_summary combines identity and credit overview data into:
    • identityVerified
    • confidenceScore
    • consumerName
    • creditScore
    • creditModel
    • openTradeLines
    • derogatoryTradeLines
    • inquiries12Months

equifax_cleaner and prompt integration

[!NOTE] generate_credit_report reads request_id from report_data and imports get_pdf_report, but the active return path only emits {"llm_summary": llm_summary, "raw_report": report_data}. The PDF fetch line is commented out, so pdf_content never contributes to the output.

The Equifax report path calls preprocess_equifax_data before the prompt is built. The cleaned payload is then passed to get_analyze_credit_report_prompt and finally to generate_llm_response, so the LLM summary always runs on sanitised report data rather than the raw upstream payload.

Japan Staff Credit Pull

backend/src/usecase/japan_staff_credit.py

backend/src/interface/japan_credit_endpoints.py

This path handles staff-initiated bureau pulls for Japan. The endpoint constructs StaffCreditPullRequest with consent_reference, operator_id, requested_at, optional device_fingerprint, and optional permissible_purpose, then calls perform_staff_credit_pull.

Visible error mapping in the endpoint:

  • ConsentNotFoundError -> 404
  • ConsentExpiredError, ConsentRevokedError, OperatorNotAllowedError -> 400
  • Any other exception -> 502

The successful response includes:

  • consent_reference
  • operator_id
  • requested_at
  • bureau_reference
  • summary
  • data_source
  • created_at
  • report

Feature Flows

TransUnion batch ingestion flow

sequenceDiagram
    participant C as Client
    participant E as TransUnion Batch Endpoint
    participant U as process_transunion_batch
    participant R as SQLTransUnionBatchRepository
    participant P as parse_line

    C->>E: POST batch ingestions
    E->>E: validate rows present
    E->>U: process_transunion_batch
    U->>R: create_ingestion received
    U->>R: update_ingestion processing
    loop each non-empty row
        U->>P: parse_line
        alt parse error
            U->>U: append issue
        else parsed row
            U->>U: collect record
        end
    end
    U->>R: add_rows
    U->>R: add_issues
    U->>R: update_ingestion completed
    R-->>U: updated ingestion
    U-->>E: result
    E-->>C: 201 response

OCR assisted upload and canonical mapping flow

sequenceDiagram
    participant C as Client
    participant U as create_signed_upload_session
    participant S as Cloud Storage
    participant F as finalize_upload_session
    participant M as map_ocr_to_canonical_credit

    C->>U: request upload session
    U->>S: generate signed PUT URL
    S-->>U: upload session response
    U-->>C: signed URL and object path

    C->>S: upload report file
    C->>F: finalize session
    F->>S: validate object size and content type
    S-->>F: object metadata
    F-->>C: active upload session

    C->>M: submit OCR payload
    M->>M: index OCR fields
    M->>M: map consumer and score fields
    M->>M: build model adapter features
    M-->>C: canonical credit report

Equifax credit report generation flow

sequenceDiagram
    participant C as Client
    participant G as generate_credit_report
    participant A as login_user
    participant B as get_credit_report
    participant N as preprocess_equifax_data
    participant P as get_analyze_credit_report_prompt
    participant L as generate_llm_response

    C->>G: submit user data
    G->>A: login_user
    A-->>G: token
    G->>B: get_credit_report
    B-->>G: report data
    G->>N: preprocess_equifax_data
    N-->>G: cleaned report
    G->>P: get_analyze_credit_report_prompt
    P-->>G: prompt
    G->>L: generate_llm_response
    L-->>G: summary
    G-->>C: llm_summary and raw_report

Japan staff credit pull flow

sequenceDiagram
    participant C as Client
    participant E as initiate_credit_pull
    participant U as perform_staff_credit_pull
    participant CR as consent repository
    participant LR as log repository
    participant B as JapanCreditBureauHTTPClient

    C->>E: submit consent reference and operator data
    E->>U: perform_staff_credit_pull
    U->>CR: verify consent
    U->>LR: write pull log
    U->>B: request bureau report
    B-->>U: bureau report
    U-->>E: result bundle
    E-->>C: pull response

State Management

TransUnion ingestion lifecycle

The TransUnion batch path stores and updates an explicit ingestion status:

  • received is written during creation
  • processing is written before row parsing begins
  • completed is written after rows and issues are persisted

OCR upload session lifecycle

The upload path uses session metadata to keep the manual upload flow consistent:

  • upload_session_id
  • user_id
  • bureau
  • country
  • object_path
  • content_type
  • file_size_bytes
  • expires_at

The canonical OCR report also stores confirmation state under report_metadata.user_confirmation with:

  • confirmed
  • confirmed_at

Equifax and Japan pull state

The Equifax report generator is a one-shot transform from fetched report data to summary output. The Japan staff pull path carries consent and pull-log state through repository calls and bureau-request logging.

Error Handling

TransUnion batch ingestion

  • Empty rows array returns HTTP 400 with At least one row is required.
  • Parse failures are converted into TransUnionBatchRowIssue entries instead of aborting the batch.
  • A missing persisted ingestion id raises RuntimeError("Ingestion must be persisted").

OCR upload

  • _get_bucket_name raises UploadConfigError when CREDIT_REPORT_UPLOAD_BUCKET is not set.
  • finalize_upload_session raises UploadValidationError when the upload session cannot be found or the object does not match expected constraints.
  • finalize_upload_session raises UploadStorageError when the object is missing or its size cannot be read.
  • Validation failures delete the uploaded blob before raising.

Equifax generation

  • The active report generation path does not catch downstream login, fetch, preprocessing, or LLM failures in this file; they propagate to the caller.

Japan staff credit pull

  • ConsentNotFoundError maps to 404.
  • ConsentExpiredError, ConsentRevokedError, and OperatorNotAllowedError map to 400.
  • Unexpected integration failures map to 502.

Dependencies

Domain and repository dependencies

  • TransUnionBatchIngestion
  • TransUnionBatchRecord
  • TransUnionBatchRowIssue
  • TransUnionBatchRepository
  • SQLTransUnionBatchRepository
  • UploadSessionRepository
  • HousingApplication is not part of this section and is not used here

Parser and transform dependencies

  • parse_line
  • load_layout_from_csv
  • FieldSpec
  • converter_from_data_type

External service dependencies

  • get_credit_report
  • login_user
  • get_pdf_report
  • preprocess_equifax_data
  • generate_llm_response
  • get_analyze_credit_report_prompt
  • JapanCreditBureauHTTPClient
  • Cloud Storage client used by create_signed_upload_session and finalize_upload_session

Pipeline consumers

  • CanonicalBureauData
  • build_credit_model_adapter_features
  • downstream credit scoring and contract layers that consume canonical bureau evidence

Testing Considerations

  • test_create_transunion_batch_ingestion verifies the batch endpoint returns 201, a completed status, and masked row output.
  • test_create_transunion_batch_ingestion_with_custom_layout verifies custom layout conversion and parsing.
  • test_process_transunion_batch_creates_summary verifies TransUnion summary generation, including creditModel, creditScore, and sanitised identifiers.
  • test_parse_line_returns_row_with_core_fields verifies parser field extraction and metric values.
  • test_parse_line_raises_for_invalid_date verifies invalid dates raise ValueError.
  • test_load_layout_from_csv_maps_core_fields_without_manual_normalised_keys verifies CSV layout loading and layout-driven parsing.
  • test_map_ocr_to_canonical_credit_includes_model_adapter verifies OCR mapping and model adapter generation.
  • test_build_credit_model_adapter_features_aggregates_tradelines verifies tradeline aggregation into adapter features.
  • test_preprocess_equifax_data_handles_tradeline_payload and test_preprocess_equifax_data_filters_structured_payload verify the Equifax cleaning step that feeds generate_credit_report.
  • test_perform_staff_credit_pull_success verifies the Japan staff pull path succeeds and logs the request.

Key Classes Reference

Class Responsibility
transunion_batch_endpoints.py Accepts TransUnion batch ingestion requests and returns processed ingestion results.
transunion_batch.py Parses and persists TransUnion batch ingestions and builds summary output.
credit_canonical_schema.py Defines the canonical bureau data shape shared by all bureau paths.
credit_adapters.py Normalises TransUnion, Equifax, and manual bureau payloads into canonical bureau data.
credit_upload.py Manages signed upload sessions and converts OCR payloads into canonical credit data.
credit_report.py Orchestrates Equifax-style report retrieval, cleaning, prompting, and summary generation.
equifax_instant_intake.py Builds Equifax credit overview and summary payloads from instant-intake responses.
japan_staff_credit.py Drives Japan staff-initiated bureau pull orchestration.
schemas.py Defines request and response schemas for batch ingestion and OCR completion flows.
test_transunion_batch_parser.py Verifies batch parser field extraction, layout loading, and invalid date handling.