Credit Scoring and Pipeline Domain - Bureau Ingestion, Uploads, and Batch Processing

Overview

This feature set is the bureau-data entry point for the credit-scoring pipeline. It covers three ingestion paths that feed the same downstream scoring model: signed manual uploads, Equifax instant pulls, and TransUnion batch files.

The implementation keeps each bureau source separate at ingress, then normalizes them into canonical payloads or canonical bureau records before scoring. That normalization is what lets the broader pipeline treat manual uploads, instant reports, and batch rows as comparable evidence for completeness checks, feature generation, and risk scoring.

Architecture Overview

flowchart TB
    subgraph UploadIngress [Manual Upload Ingestion]
        UploadSessionCreate[create_signed_upload_session]
        ObjectStorage[Object storage blob]
        UploadFinalize[finalize_upload_session]
        OCRMap[map_ocr_to_canonical_credit]
        ModelAdapter[build_credit_model_adapter_features]
    end

    subgraph EquifaxFlow [Equifax Instant Intake]
        EquifaxEndpoint[POST api equifax instant intake]
        EquifaxUseCase[perform_equifax_instant_intake]
        EquifaxClient[EquifaxClient]
        EquifaxCleaner[preprocess_equifax_data]
        EquifaxSummary[_build_summary]
    end

    subgraph TransUnionFlow [TransUnion Batch Processing]
        TransUnionEndpoint[POST api transunion batch ingestions]
        BatchUseCase[process_transunion_batch]
        BatchParser[parse_line]
        BatchRepo[TransUnionBatchRepository]
        BatchSummary[_build_summary]
    end

    subgraph JapanFlow [Japan Staff Pull]
        StaffUseCase[perform_staff_credit_pull]
        ConsentRepo[StaffCreditConsentRepository]
        PullLogRepo[StaffCreditPullLogRepository]
        BureauClient[Bureau client]
    end

    subgraph ScoringBridge [Canonical Scoring Bridge]
        BureauAdapter[BureauAdapter]
        CanonicalData[CanonicalBureauData]
        PipelineGate[credit_pipeline_validation]
        RuntimeScorer[credit_score_runtime]
        CreditModel[credit_model]
    end

    UploadSessionCreate --> ObjectStorage
    ObjectStorage --> UploadFinalize
    UploadFinalize --> OCRMap
    OCRMap --> ModelAdapter
    ModelAdapter --> CanonicalData

    EquifaxEndpoint --> EquifaxUseCase
    EquifaxUseCase --> EquifaxClient
    EquifaxUseCase --> EquifaxCleaner
    EquifaxUseCase --> EquifaxSummary
    EquifaxUseCase --> CanonicalData

    TransUnionEndpoint --> BatchUseCase
    BatchUseCase --> BatchParser
    BatchUseCase --> BatchRepo
    BatchUseCase --> BatchSummary
    BatchUseCase --> BureauAdapter
    BureauAdapter --> CanonicalData

    StaffUseCase --> ConsentRepo
    StaffUseCase --> PullLogRepo
    StaffUseCase --> BureauClient
    StaffUseCase --> CanonicalData

    CanonicalData --> PipelineGate
    PipelineGate --> RuntimeScorer
    RuntimeScorer --> CreditModel

Upload Session Lifecycle

File paths: backend/src/usecase/credit_upload.py, backend/tests/usecase/test_credit_upload.py

Manual bureau uploads enter the platform through a signed upload session. The session records the declared bureau, country, object path, file size, and content type before any file is accepted.

Upload Session Abstraction

The upload path uses an UploadSession record to bind the user, bureau, and exact storage object that will hold the document.

Property Type Description
upload_session_id str Unique upload-session identifier.
user_id str Owner of the upload session.
bureau str Bureau label used in the object path and later canonical metadata.
country str Country code recorded with the session.
object_path str Storage path for the uploaded file.
content_type str Declared MIME type used for validation.
file_size_bytes int Declared file size limit for the upload.
expires_at datetime Session expiration timestamp.

The signed-session response exposes the storage details needed by the uploader.

Property Type Description
upload_session_id str Session identifier returned to the caller.
expires_at datetime Upload window expiration.
upload_url str Signed PUT URL for object storage.
object_path str Final object path to upload to.
content_type str MIME type required by the signed request.
file_size_bytes int Maximum allowed size for the object.

Upload Use-Case Methods

Method Description
create_signed_upload_session Creates a new upload session, persists it through UploadSessionRepository, and generates a signed storage URL for the file.
finalize_upload_session Reloads the stored object and validates size, content type, and maximum upload constraints before allowing downstream OCR mapping.
map_ocr_to_canonical_credit Converts OCR-extracted fields into a canonical credit-report structure with provenance and model-adapter payloads.
build_credit_model_adapter_features Aggregates OCR tradelines into the compact adapter feature vector used by the scoring model.
_extract_tables Walks document tables and extracts header/body cells from page objects.

Signed Upload Flow

  1. create_signed_upload_session generates upload_session_id, computes expires_at, and stores an UploadSession.
  2. The file is uploaded directly to object storage using the returned signed URL.
  3. finalize_upload_session fetches the session by upload_session_id, user_id, and bureau.
  4. The blob size and content_type are compared against the declared values.
  5. map_ocr_to_canonical_credit converts validated OCR output into a canonical report with:
    • report_metadata
    • consumer
    • scores
    • summary
    • model_adapter
    • raw_ocr

OCR-to-Canonical Output

map_ocr_to_canonical_credit keeps the uploaded document as traceable evidence and also builds a structured model_adapter payload. The adapter output is what the rest of the pipeline uses for bureau-derived features.

build_credit_model_adapter_features is verified by tests to aggregate tradelines into:

  • no_of_unsecured_loans
  • value_of_unsecured_loans
  • no_of_major_loans
  • value_of_major_loans
  • years_since_first_major_loan
  • tradeline_count

and to retain normalized_tradelines entries.

Upload Flow

sequenceDiagram
    participant U as Uploader
    participant S as signed session use case
    participant O as Object storage
    participant C as OCR mapper
    participant P as scoring bridge

    U->>S: create_signed_upload_session
    S->>S: persist UploadSession
    S-->>U: upload_url and object_path
    U->>O: PUT document
    U->>S: finalize_upload_session
    S->>O: load blob metadata
    S-->>U: validated session
    U->>C: map_ocr_to_canonical_credit
    C-->>P: canonical report and model_adapter

Equifax Instant Intake

File paths: backend/src/usecase/equifax_instant_intake.py, backend/src/interface/transunion_batch_endpoints.py does not apply here, backend/tests/interface/test_equifax_endpoints.py, backend/tests/usecase/test_equifax_instant_intake.py, backend/tests/infra/external_apis/test_equifax_cleaner.py

Equifax intake is the synchronous bureau path. It creates a processing record, fetches identity and credit data from the Equifax client, stores the raw payloads, then returns a normalized summary for scoring.

Request and Result Models

EquifaxInstantIntakeRequest

Property Type Description
phone str Applicant phone number used for the instant ID lookup.
date_of_birth date Date of birth passed to Equifax.
permissible_purpose str Consent purpose forwarded to the bureau client.

EquifaxInstantIntakeRecord

Property Type Description
id `int \ None` Persistence identifier.
phone str Lookup phone number.
date_of_birth date Applicant birth date.
status str Processing state.
permissible_purpose str Consent purpose used for the pull.
consent_captured_at `datetime \ None` Captured consent timestamp.
instant_id_reference `str \ None` Equifax instant-ID reference.
credit_report_reference `str \ None` Equifax credit-report reference.
identity_payload `dict[str, Any] \ None` Raw instant-ID payload.
credit_report_payload `dict[str, Any] \ None` Raw credit-report payload.
error_message `str \ None` Captured error message for failed pulls.
created_at `datetime \ None` Record creation timestamp.
updated_at `datetime \ None` Last update timestamp.

EquifaxInstantIntakeResult

Property Type Description
record EquifaxInstantIntakeRecord Persisted intake record after status update.
identity_profile `dict[str, Any] \ None` Normalized identity profile.
credit_report_overview `dict[str, Any] \ None` Normalized credit-overview fields.
summary dict[str, Any] Combined identity and credit summary.
sanitized_identity_payload dict[str, Any] Redacted identity payload for safe storage/return.
sanitized_credit_report_payload dict[str, Any] Redacted credit payload for safe storage/return.
data_source str Source tag returned by the bureau client.

Equifax Use-Case Methods

Method Description
perform_equifax_instant_intake Orchestrates the full Equifax pull, updates status, sanitizes payloads, and returns the normalized result.
_build_summary Merges identity verification state with bureau score and trade-line summary data.
_build_credit_overview Extracts score, score model, reason codes, balances, trade-line counts, and message list from the credit payload.

Equifax Data Flow

perform_equifax_instant_intake does the following:

  1. Calls _ensure_consent_timestamp to validate or resolve the consent time.
  2. Creates an EquifaxInstantIntakeRecord with status="processing".
  3. Calls equifax_client.fetch_instant_id(...).
  4. Calls equifax_client.fetch_credit_report(...) using the returned identity payload.
  5. On EquifaxClientError, marks the record failed and records the error message.
  6. Updates the record with request references and full payloads.
  7. Builds:
    • identity_profile
    • credit_overview
    • summary
  8. Returns sanitized copies of both raw payloads for safe downstream use.

Equifax Payload Cleaner

File path: backend/src/infra/external_apis/equifax_cleaner.py

preprocess_equifax_data is the normalization step that removes unsupported fields before the report is analyzed or stored.

Verified behavior from tests:

  • Preserves top-level fields such as CREDIT_AS_OF_DATE.
  • Whitelists keys under PPPCL, EADS232, and EADS76.
  • Normalizes bureau history into creditHistory.
  • Maps trade-line details such as:
    • accountType
    • delinquency
    • payment-history and narrative fields
  • Retains inquiries as provided.

Equifax API

Perform Equifax Instant Intake

{
    "title": "Perform Equifax Instant Intake",
    "description": "Creates an intake record, fetches instant identity and credit data from Equifax, persists the raw payloads, and returns a normalized summary.",
    "method": "POST",
    "baseUrl": "<BackendApiBaseUrl>",
    "endpoint": "/api/equifax/instant-intake",
    "headers": [
        {
            "key": "Content-Type",
            "value": "application/json",
            "required": true
        }
    ],
    "queryParams": [],
    "pathParams": [],
    "bodyType": "json",
    "requestBody": "{\n    \"phone\": \"+1 (555) 123-4567\",\n    \"date_of_birth\": \"1990-04-12\",\n    \"consent_captured_at\": \"2024-01-01T00:00:00Z\",\n    \"permissible_purpose\": \"credit_application\"\n}",
    "formData": [],
    "rawBody": "",
    "responses": {
        "201": {
            "description": "Completed intake",
            "body": "{\n    \"status\": \"completed\",\n    \"data_source\": \"equifax\",\n    \"identity_profile\": {\n        \"requestId\": \"IID-TEST-001\",\n        \"matchIndicator\": \"MATCHED\",\n        \"confidenceScore\": 0.97,\n        \"firstName\": \"Jane\",\n        \"lastName\": \"Doe\",\n        \"fullName\": \"Jane Q. Doe\",\n        \"dateOfBirth\": \"1990-04-12\",\n        \"ssn\": {\n            \"last4\": \"6789\",\n            \"full\": \"123-45-6789\"\n        },\n        \"addresses\": [\n            {\n                \"line1\": \"123 Market St\",\n                \"city\": \"San Francisco\",\n                \"state\": \"CA\",\n                \"postalCode\": \"94105\",\n                \"country\": \"US\"\n            }\n        ],\n        \"phones\": [\n            {\n                \"number\": \"+1 (555) 123-4567\",\n                \"type\": \"MOBILE\",\n                \"lastSeen\": \"2024-03-01\"\n            }\n        ],\n        \"alerts\": [],\n        \"notes\": [\n            \"Test fixture response\"\n        ]\n    },\n    \"credit_report_overview\": {\n        \"model\": \"FICO Score 9\",\n        \"score\": 742,\n        \"reasonCodes\": [\n            \"05\",\n            \"10\"\n        ],\n        \"totalTradeLines\": 12,\n        \"openTradeLines\": 9,\n        \"revolvingBalance\": 12540,\n        \"installmentBalance\": 22100,\n        \"derogatoryTradeLines\": 0,\n        \"inquiries12Months\": 1,\n        \"messages\": [\n            \"Test fixture credit report\"\n        ]\n    },\n    \"summary\": {\n        \"identityVerified\": true,\n        \"confidenceScore\": 0.97,\n        \"consumerName\": \"Jane Doe\",\n        \"creditScore\": 742,\n        \"creditModel\": \"FICO Score 9\",\n        \"openTradeLines\": 9,\n        \"derogatoryTradeLines\": 0,\n        \"inquiries12Months\": 1\n    }\n}"
        }
    }
}

Equifax Intake Flow

sequenceDiagram
    participant C as Client
    participant A as Equifax API
    participant U as perform_equifax_instant_intake
    participant R as EquifaxIntakeRepository

    C->>U: request payload
    U->>R: create processing record
    U->>A: fetch_instant_id
    A-->>U: identity payload
    U->>A: fetch_credit_report
    A-->>U: credit payload
    U->>R: update_status completed or failed
    U-->>C: result with summary and sanitized payloads

TransUnion Batch Processing

File paths: backend/src/usecase/transunion_batch.py, backend/src/interface/transunion_batch_endpoints.py, backend/tests/interface/test_transunion_batch_endpoints.py, backend/tests/infra/importers/test_transunion_batch_parser.py, backend/tests/infra/mysql/test_transunion_batch_repository.py, backend/tests/usecase/test_transunion_batch.py

The TransUnion path is batch-oriented. It accepts a file name, checksum, and a row set, parses each row, persists both clean rows and row-level issues, then returns a summary that the scoring pipeline can ingest.

Batch Record Shapes

TransUnionBatchIngestion

Property Type Description
id `int \ None` Persistence identifier.
file_name str Source batch file name.
checksum str File checksum used for idempotency or audit.
status str Ingestion lifecycle state.
total_rows int Count of input rows.
credit_as_of_date `date \ None` Credit-as-of date attached to the batch.
metadata dict[str, object] Arbitrary batch metadata.

TransUnionBatchRecord

Property Type Description
id `int \ None` Persistence identifier.
ingestion_id int Parent ingestion identifier.
perm_id str Person identifier from the batch row.
credit_as_of_date `date \ None` Row-level credit-as-of date.
consumer_id `str \ None` Consumer identifier from the row.
sequence_number `str \ None` Sequence value from the row.
metrics dict[str, object] Normalized metric dictionary built by the parser.

TransUnionBatchRowIssue

Property Type Description
id `int \ None` Persistence identifier.
ingestion_id int Parent ingestion identifier.
row_number int One-based row number in the source file.
message str Parsing error message.
raw_row str Original unparsed row text.

ProcessTransUnionBatchResult

Property Type Description
ingestion TransUnionBatchIngestion Final ingestion record after processing.
summary dict[str, object] Aggregated score summary.
sanitized_rows list[dict[str, object]] Masked, API-safe row representations.
issues list[TransUnionBatchRowIssue] Row-level parse issues.

TransUnion Use-Case Methods

Method Description
process_transunion_batch Parses rows, stores batch records and issues, computes the summary, and closes the ingestion.
_build_summary Computes average, min, max, and last credit scores from parsed records.
_build_sanitized_row Masks identifiers and returns API-safe row data with metrics.

Batch Processing Flow

process_transunion_batch runs in a fixed sequence:

  1. Materializes the input rows into a list.
  2. Creates a TransUnionBatchIngestion with status="received".
  3. Updates the ingestion to status="processing".
  4. For each non-empty row:
    • Calls parse_line.
    • On parse failure, stores a TransUnionBatchRowIssue.
    • On success, creates a TransUnionBatchRecord.
  5. Persists valid rows and issues.
  6. Builds sanitized_rows using _build_sanitized_row.
  7. Builds summary using _build_summary.
  8. Marks the ingestion completed and records counts.
  9. Returns ProcessTransUnionBatchResult.

Batch Summary Semantics

The summary carries the fields that the broader scoring layer consumes:

  • dataSource: always transunion_batch
  • totalRows
  • creditModel: VantageScore 4.0 when vantage scores are present
  • averageCreditScore
  • minCreditScore
  • maxCreditScore
  • creditScore
  • truVisionNewAccountScore when available

Row Sanitization

_build_sanitized_row preserves the row-level metrics but masks identifiers:

  • permId is masked
  • consumerId is masked
  • sequenceNumber is passed through
  • metrics are copied as dictionaries so API consumers do not receive the original mutable row object

Parser Coverage

backend/tests/infra/importers/test_transunion_batch_parser.py shows that parse_line returns rows with:

  • perm_id
  • consumer_id
  • sequence_number
  • credit_as_of_date
  • metrics["vantage_score"]["value"]
  • metrics["vantage_score"]["description"]
  • metrics["truvision_new_account_score"]["value"]

TransUnion API

Create TransUnion Batch Ingestion

{
    "title": "Create TransUnion Batch Ingestion",
    "description": "Parses TransUnion batch rows, persists the ingestion and row issues, and returns the sanitized batch summary.",
    "method": "POST",
    "baseUrl": "<BackendApiBaseUrl>",
    "endpoint": "/api/transunion/batch-ingestions",
    "headers": [
        {
            "key": "Content-Type",
            "value": "application/json",
            "required": true
        }
    ],
    "queryParams": [],
    "pathParams": [],
    "bodyType": "json",
    "requestBody": "{\n    \"file_name\": \"tu_batch.txt\",\n    \"checksum\": \"chk-1\",\n    \"credit_as_of_date\": \"2023-04-30\",\n    \"rows\": [\n        \"perm_id|credit_as_of_date|consumer_id|sequence_number|vantage_score|truvision_new_account_score\"\n    ],\n    \"metadata\": {\n        \"triggeredBy\": \"test\"\n    }\n}",
    "formData": [],
    "rawBody": "",
    "responses": {
        "201": {
            "description": "Batch ingestion completed",
            "body": "{\n    \"status\": \"completed\",\n    \"summary\": {\n        \"dataSource\": \"transunion_batch\",\n        \"totalRows\": 1,\n        \"creditModel\": \"VantageScore 4.0\",\n        \"averageCreditScore\": 690,\n        \"minCreditScore\": 690,\n        \"maxCreditScore\": 690,\n        \"creditScore\": 690,\n        \"truVisionNewAccountScore\": 210\n    },\n    \"sanitized_rows\": [\n        {\n            \"creditAsOfDate\": \"2023-04-30\",\n            \"permId\": \"******0002\",\n            \"consumerId\": \"******0101\",\n            \"sequenceNumber\": \"01\",\n            \"metrics\": {\n                \"vantage_score\": {\n                    \"value\": 690,\n                    \"description\": \"VantageScore 4.0\"\n                },\n                \"truvision_new_account_score\": {\n                    \"value\": 210,\n                    \"description\": \"TruVision New Account Score\"\n                }\n            }\n        }\n    ],\n    \"issues\": []\n}"
        }
    }
}

Batch Ingestion Flow

sequenceDiagram
    participant C as Client
    participant E as TransUnion endpoint
    participant U as process_transunion_batch
    participant P as parse_line
    participant R as TransUnionBatchRepository

    C->>E: POST batch payload
    E->>U: call use case
    U->>R: create_ingestion received
    U->>R: update_ingestion processing
    loop each row
        U->>P: parse_line
        alt parse failure
            U->>R: add_issues
        else parse success
            U->>R: add_rows
        end
    end
    U->>R: update_ingestion completed
    U-->>E: result summary and sanitized rows
    E-->>C: 201 response

Japan Staff Credit Pull

File paths: backend/src/usecase/japan_staff_credit.py, backend/tests/usecase/test_japan_staff_credit.py

This use case handles operator-initiated bureau pulls in Japan. The code path is consent-driven and logs the pull outcome for auditability.

Public Methods

Method Description
register_staff_credit_consent Registers a staff consent entry for later bureau access.
perform_staff_credit_pull Validates consent and operator authorization, performs the bureau request, and records the pull outcome.

Observed Behavior

The tests prove the following workflow:

  • perform_staff_credit_pull requires a stored consent reference.
  • operator_id must be authorized for the consent.
  • The bureau client is called with permissible_purpose="tenancy_screening".
  • Successful pulls write a StaffCreditPullLog with status="succeeded".
  • The pull result includes the bureau summary, such as {"score": 712, "grade": "A"} in the fixture.

Error Conditions

The use case imports and surfaces these domain exceptions:

  • ConsentNotFoundError
  • ConsentExpiredError
  • ConsentRevokedError
  • BureauRequestError
  • OperatorNotAllowedError

Japan Staff Pull Flow

sequenceDiagram
    participant O as Operator
    participant U as perform_staff_credit_pull
    participant CR as StaffCreditConsentRepository
    participant LR as StaffCreditPullLogRepository
    participant B as Bureau client

    O->>U: consent_reference and operator_id
    U->>CR: load consent
    U->>U: validate consent and operator
    U->>B: request_credit_report
    B-->>U: BureauCreditReport
    U->>LR: store succeeded log
    U-->>O: report summary

Canonical Bureau Normalization and Scoring Reconciliation

File paths: backend/src/ml/credit_canonical_schema.py, backend/src/ml/credit_adapters.py, backend/src/ml/credit_variable_map.py, backend/src/ml/credit_model.py, backend/src/ml/credit_pipeline_validation.py, backend/src/usecase/credit_score_runtime.py

This is the bridge between bureau ingestion and the scoring model. The platform does not score raw TransUnion or Equifax payloads directly; it normalizes them into CanonicalBureauData, converts those fields into feature columns, and only then merges them into the final scoring input.

Canonical Bureau Data

CanonicalBureauData

Property Type Description
provider str Bureau source label such as transunion, equifax, manual_upload, or global.
score_value `float \ None` Raw score in the source provider scale.
score_scale_min float Minimum score scale value.
score_scale_max float Maximum score scale value.
score_provider str Source-specific score model label.
utilization_ratio `float \ None` Overall revolving utilization.
delinquencies_24m int Count of delinquency events in the last 24 months.
worst_dpd_24m int Worst days-past-due bucket in the last 24 months.
credit_age_months `float \ None` Age of the newest trade.
credit_age_months_oldest `float \ None` Age of the oldest trade.
inquiries_12m int Bureau inquiries in the last 12 months.
new_accounts_12m int New accounts in the last 12 months.
collections_flag bool Collection-event flag.
collections_count int Collection count.
default_flag bool Default-event flag.
report_recency_days float Days since the report was pulled.
parse_confidence float Parsing confidence from OCR or ingest.
fraud_risk_flag bool Fraud risk indicator.
aml_risk_flag bool AML risk indicator.
estimated_income_score `float \ None` TransUnion estimated income score.
revolving_util_patterns dict[str, float] Trended revolving utilization buckets.
bankcard_util_patterns dict[str, float] Trended bankcard utilization buckets.
trade_age_months dict[str, float] Trade-age buckets by product class.
loan_balances dict[str, float] Loan-balance buckets by product class.
raw dict[str, Any] Original source payload retained for traceability.

Bureau Adapter Methods

Method Description
BureauAdapter.from_transunion_row Normalizes a TransUnion batch row into CanonicalBureauData.
BureauAdapter.from_equifax_payload Normalizes an Equifax instant-intake payload into CanonicalBureauData.
BureauAdapter.from_manual_report Normalizes a manually uploaded or OCR-extracted bureau report into CanonicalBureauData.

TransUnion Adapter Behavior

BureauAdapter.from_transunion_row accepts both renamed and raw TransUnion field names, and the renamed values win when both are present.

Verified behavior includes:

  • VantageScore maps to score_value
  • Estimated_Income_Score maps to estimated_income_score
  • Months_Since_Oldest_Trade_Opened maps to credit_age_months_oldest
  • Delinquency windows are converted into worst_dpd_24m
  • revolving_util_patterns, bankcard_util_patterns, trade_age_months, and loan_balances are populated from both renamed and raw source codes
  • raw preserves extra fields such as extra_field

Equifax Adapter Behavior

BureauAdapter.from_equifax_payload reads the credit_profile envelope and maps:

  • risk_score.value to score_value
  • risk_score.scale_min and risk_score.scale_max to the score range
  • tradelines to delinquency counts and worst DPD
  • utilization_ratio or revolving_utilization to utilization_ratio
  • inquiries_12m or inquiries to inquiries_12m
  • collections_count, default_flag, recency_days, parse_confidence, fraud_risk_flag, aml_risk_flag

Manual Report Adapter Behavior

BureauAdapter.from_manual_report is the path used after OCR extraction. It reads a generic report dictionary with fields such as:

  • score_value or score
  • score_scale_min
  • score_scale_max
  • score_provider
  • recency_days
  • parse_confidence

Model Feature Bridge

The bureau-side fields are reconciled into model features through credit_variable_map.py and credit_model.py. The feature set includes:

  • bureau_available_flag
  • bureau_score_raw
  • bureau_score_value
  • bureau_utilization
  • bureau_utilization_ratio
  • bureau_delinquency_24m
  • bureau_delinquencies_24m
  • bureau_worst_dpd_24m
  • bureau_credit_age_months
  • bureau_credit_age_oldest_months
  • bureau_inquiries_12m
  • bureau_new_accounts_12m
  • bureau_collections_flag
  • bureau_collections_count
  • bureau_default_flag
  • bureau_defaults_flag
  • bureau_report_recency_days
  • bureau_parse_confidence
  • bureau_strength
  • estimated_income_score
  • trended buckets such as rev_util_*, bkc_util_*, trade_age_*, and loan_balance_*

credit_score_runtime.py then reads bureau evidence from application.connected_accounts, specifically from:

  • bureau_report
  • manual_credit_report
  • bureau

and converts that data into runtime features such as:

  • bureau_available_flag
  • bureau_score_raw
  • bureau_score_value
  • bureau_report_recency_days
  • bureau_parse_confidence
  • bureau_strength
  • bureau_utilization
  • bureau_delinquency_24m
  • bureau_worst_dpd_24m
  • bureau_credit_age_months
  • bureau_credit_age_oldest_months
  • bureau_inquiries_12m
  • bureau_new_accounts_12m
  • bureau_collections_flag
  • bureau_collections_count
  • bureau_default_flag
  • bureau_defaults_flag
  • bureau_available_flag

Pipeline Gating

credit_pipeline_validation.py applies deterministic gating before scoring.

Relevant constants:

  • MIN_DAYS_COVERED
  • BUREAU_MAX_AGE_DAYS
  • MIN_COMPLETENESS_SCORE_TO_PASS
  • _BUREAU_REQUIRED_PRODUCTS

Classification flags:

  • Decline: KYC_NOT_PASSED, FRAUD_RISK_DETECTED, AML_RISK_DETECTED
  • Refer: BUREAU_REQUIRED_BUT_ABSENT, NO_BANKING_DATA, INSUFFICIENT_HISTORY

This means bureau data can be present but still not eligible for scoring if the completeness gate fails or the product requires bureau data and none is available.

Error Handling

The bureau ingestion paths fail differently depending on the source and stage.

Error or Outcome Where It Appears Effect
UploadValidationError finalize_upload_session Rejects invalid sessions, size mismatches, or content-type mismatches.
UploadStorageError finalize_upload_session Rejects missing blobs or missing blob size metadata.
EquifaxClientError perform_equifax_instant_intake Marks the intake record failed and captures the error text.
RuntimeError perform_equifax_instant_intake, process_transunion_batch Raised when a record is expected to be persisted but is still missing an id.
parsing exception process_transunion_batch Converted into a TransUnionBatchRowIssue instead of stopping the whole batch.
ConsentNotFoundError Japan staff pull Stops the pull when the consent reference is unknown.
ConsentExpiredError Japan staff pull Stops the pull when the consent is expired.
ConsentRevokedError Japan staff pull Stops the pull when consent is no longer valid.
OperatorNotAllowedError Japan staff pull Stops the pull when the operator is not allowed to use the consent.
BureauRequestError Japan staff pull Captures bureau request failures.

Testing Considerations

[!NOTE] process_transunion_batch isolates row-level parse failures into TransUnionBatchRowIssue records and still completes the ingestion for all remaining valid rows.

The supplied tests cover the key bureau-entry behaviors:

  • test_map_ocr_to_canonical_credit_includes_model_adapter
  • test_build_credit_model_adapter_features_aggregates_tradelines
  • test_preprocess_equifax_data_filters_structured_payload
  • test_preprocess_equifax_data_handles_tradeline_payload
  • test_perform_equifax_instant_intake_generates_summary
  • test_equifax_instant_intake_endpoint
  • test_process_transunion_batch_creates_summary
  • test_create_transunion_batch_ingestion
  • test_parse_line_returns_row_with_core_fields
  • test_perform_staff_credit_pull_success
  • test_perform_staff_credit_pull_rejects_unregistered_operator
  • TestBureauAdapterTransUnion
  • TestBureauAdapterEquifax

These tests collectively prove that:

  • manual upload OCR output is canonicalized before scoring,
  • Equifax instant pulls are summarized and sanitized,
  • TransUnion batch rows are parsed, persisted, and summarized,
  • bureau adapters normalize multiple provider shapes into one schema,
  • and the runtime scoring pipeline receives bureau-derived evidence in a format the model can consume.

Dependencies

Core Application Dependencies

  • FastAPI for the batch HTTP endpoint layer.
  • SQLAlchemy Session for the TransUnion repository path.
  • UploadSessionRepository for manual-upload session persistence.
  • EquifaxIntakeRepository for instant-intake record lifecycle management.
  • TransUnionBatchRepository for batch ingestion persistence.
  • EquifaxClient for bureau pulls.
  • FieldSpec, converter_from_data_type, and parse_line for TransUnion parsing.
  • preprocess_equifax_data for Equifax payload sanitization.
  • credit_canonical_schema, credit_adapters, credit_model, and credit_pipeline_validation for the scoring bridge.

External Services

  • Object storage for signed uploads and blob validation.
  • Equifax bureau API via EquifaxClient.
  • TransUnion batch source files via the importer and repository path.

Logging

backend/src/ml/credit_adapters.py creates a module logger with logging.getLogger(__name__) for adapter-side diagnostics.

Key Classes Reference

Class Responsibility
UploadSession Tracks a signed upload request and its storage constraints.
UploadSessionResponse Returns the signed URL and storage metadata for the upload client.
EquifaxInstantIntakeRecord Persists the lifecycle and payloads for an Equifax instant pull.
EquifaxInstantIntakeRequest Captures the request parameters for an Equifax instant intake.
EquifaxInstantIntakeResult Packages the persisted record, normalized profiles, summary, and sanitized payloads.
TransUnionBatchIngestion Represents a batch file lifecycle from receipt to completion.
TransUnionBatchRecord Stores one parsed TransUnion row and its normalized metrics.
TransUnionBatchRowIssue Captures a row-level parse failure without aborting the batch.
ProcessTransUnionBatchResult Returns the completed ingestion, summary, sanitized rows, and issues.
CanonicalBureauData Provides the provider-agnostic bureau schema used by scoring and gating.
BureauAdapter Normalizes TransUnion, Equifax, and manual bureau payloads into canonical data.