Engineering Reference, Workspace Memory, and Supporting Documents - Utility Scripts, Validation Helpers, and Local Data Generation Assets

Overview

This part of the repository contains the small command-line and test-only helpers used to create sample housing applications, clear application data from a database, and generate deterministic fixture payloads for local validation. The utilities are intentionally narrow: they either exercise the housing-application API with realistic values or build in-memory objects that drive scoring and endpoint tests.

The operational distinction is important. backend/src/scripts/create_test_application.py is a data-seeding helper that talks to the running backend and creates real application records. backend/src/scripts/clear_housing_applications.py is a destructive database maintenance script that removes all housing applications from the configured database. The remaining helpers are test fixtures and generators that stay inside the local test harness.

Script and Helper Map

File Role Primary Effect Safe for Local Use Safe for Shared or Deployment Environments
backend/src/scripts/create_test_application.py API seeding helper Creates a test housing application, then reads it back and prints a compact JSON summary Yes Only with an intentionally targeted backend
backend/src/scripts/clear_housing_applications.py Database cleanup helper Deletes every housing application in the configured database Yes, on disposable databases No
backend/tests/usecase/test_credit_scoring.py Test data builder Builds a baseline HousingApplication for credit score tests Yes N/A
backend/tests/usecase/test_credit_score_runtime.py Test data builder Builds a richer HousingApplication with Plaid-like connected accounts and transactions Yes N/A
backend/tests/usecase/test_dummy_account_detection.py Test data builder Builds a HousingApplication for dummy-account heuristic tests Yes N/A
backend/tests/ml/test_pipeline_validation.py Validation fixture builders Builds canonical bureau and banking objects for gating tests Yes N/A
backend/tests/interface/test_application_endpoint.py Payload builder Builds a realistic application payload for endpoint tests Yes N/A
backend/tests/test_applicants.py Test harness helpers Creates isolated SQLAlchemy engines, sessions, and FastAPI app instances for endpoint tests Yes N/A

Execution Overview

flowchart LR
    subgraph LocalScripts[Local Utility Scripts]
        CreateTestApp[create_test_application py]
        ClearApplications[clear_housing_applications py]
    end

    subgraph TestData[Local Test Data Builders]
        CreditScoringBuilder[test_credit_scoring build_application]
        RuntimeBuilder[test_credit_score_runtime build_application]
        DummyAccountBuilder[test_dummy_account_detection build_application]
        PipelineBuilders[test_pipeline_validation bureau and banking]
        EndpointPayloadBuilder[test_application_endpoint build_payload]
    end

    subgraph BackendServices[Backend Services]
        ApplicationsAPI[Housing applications API]
        HousingRepo[SQLAlchemy housing application repository]
        TestHarness[FastAPI and SQLite test harness]
    end

    CreateTestApp --> ApplicationsAPI
    ApplicationsAPI --> CreateTestApp

    ClearApplications --> HousingRepo
    HousingRepo --> ClearApplications

    CreditScoringBuilder --> TestHarness
    RuntimeBuilder --> TestHarness
    DummyAccountBuilder --> TestHarness
    PipelineBuilders --> TestHarness
    EndpointPayloadBuilder --> TestHarness

Utility Scripts

backend/src/scripts/create_test_application.py

[!WARNING] backend/src/scripts/clear_housing_applications.py performs a full delete against the active database session. It is only appropriate for local, throwaway, or otherwise fully disposable databases.

File path: backend/src/scripts/create_test_application.py

This script creates a test housing application through the backend HTTP API and then fetches the created record by ID. It is designed to generate a realistic applicant payload, exercise backend validation and provisioning paths, and print a concise result that includes the application ID, resolved credit score, login credentials, and the detail URL.

Functions

Method Description
_default_payload Builds the JSON body used for the initial application POST, including applicant identity, residency, housing preferences, income fields, policy consent, and a sample connected account.
_build_headers Builds request headers for JSON submission and attaches Authorization: Bearer <token> when APPLICATIONS_API_TOKEN is present.
_request_json Sends an HTTP request, decodes the JSON response, and raises a RuntimeError for transport failures, HTTP errors, or invalid JSON responses.
main Parses CLI arguments, posts a test application, reads the created record back, and prints a JSON summary to stdout.

CLI Arguments

Argument Description Default
--base-url Backend API base URL APPLICATIONS_API_BASE_URL or http://localhost:8000/api
--email Applicant email address [email protected]
--full-name Applicant full name Test Applicant

Environment Assumptions

Variable Purpose
APPLICATIONS_API_BASE_URL Overrides the backend base URL used for the POST and GET calls
APPLICATIONS_API_TOKEN Optional bearer token sent as an Authorization header

What It Generates

  • A fully populated application payload with:
    • email, phone, country, property type, purchase intent, budget range, savings, income, currency, employment status
    • financing_consent, agree_policy, receive_updates
    • a sample connected_accounts object
  • A printed JSON result containing:
    • application_id
    • credit_score
    • login_credentials
    • detail_url

Runtime Behavior

The script first posts to the applications endpoint, then immediately fetches the created application by ID. The returned credit_score prefers the detail response and falls back to the creation response when needed. If the backend returns malformed JSON or the request fails, the script stops with a RuntimeError instead of continuing with partial data.

Safety Profile

  • Safe locally: yes
  • Safe in staging: only when the target backend is meant to receive seeded test applicants
  • Safe in production: only if intentionally creating records in the live system

Sequence of Work

sequenceDiagram
    participant O as Operator
    participant S as main
    participant A as Backend API
    participant R as Request helper

    O->>S: Run create_test_application py
    S->>S: Build default payload and headers
    S->>R: POST application request
    R->>A: Submit applicant record
    A-->>R: Created application JSON
    R-->>S: Created application data
    S->>R: GET application by id
    R->>A: Fetch application detail
    A-->>R: Application detail JSON
    R-->>S: Detail response
    S-->>O: Print summary JSON

[!NOTE] The backend create flow triggered by this script also performs downstream work visible in src/interface/http_endpoints.py: it validates required fields, merges metadata into connected accounts, submits the housing application, attempts open-banking consent sync, attempts credential provisioning, and publishes an application_created event when the application has both an ID and timestamp.


backend/src/scripts/clear_housing_applications.py

File path: backend/src/scripts/clear_housing_applications.py

This script removes every housing application from the database configured for the backend session. It is a maintenance helper for resetting local or test data, not a selective cleanup tool.

Functions

Method Description
_build_argument_parser Defines the --yes flag and the command description.
_confirm_deletion Prompts for an interactive confirmation token and only proceeds when the response is exactly delete.
main Opens a database session, constructs SQLAlchemyHousingApplicationRepository, deletes all housing applications, and prints the deletion count.

CLI Arguments

Argument Description Default
--yes Skips the interactive confirmation prompt False

Environment Assumptions

Dependency Purpose
SessionLocal Provides the SQLAlchemy session bound to the configured backend database
SQLAlchemyHousingApplicationRepository Performs the actual full-table delete
Database configuration Determines which persistent store is cleared

What It Generates or Removes

  • Removes all housing application records
  • Prints Deleted N housing application(s).
  • Emits Aborted. No records were deleted. when confirmation is not provided

Runtime Behavior

The script prompts before deletion unless --yes is passed. If standard input is unavailable, _confirm_deletion returns False and the script exits without deleting anything. The database session is always closed in a finally block.

Safety Profile

  • Safe locally: yes, only against disposable databases
  • Safe in staging: no, unless the environment is intentionally resettable
  • Safe in production: no

Sequence of Work

sequenceDiagram
    participant O as Operator
    participant S as main
    participant P as confirm prompt
    participant DB as Database session
    participant R as SQLAlchemy repository

    O->>S: Run clear_housing_applications py
    S->>P: Ask for delete confirmation
    P-->>S: delete or abort
    alt confirmed
        S->>DB: Open SessionLocal
        S->>R: delete_all
        R-->>S: Deleted count
        S-->>O: Print deletion summary
    else aborted
        S-->>O: Print abort message
    end

Validation Helpers and Local Data Generation Assets

backend/tests/usecase/test_credit_scoring.py

File path: backend/tests/usecase/test_credit_scoring.py

This helper builds a baseline HousingApplication object for heuristic credit scoring tests. It provides a predictable applicant profile with empty connected accounts by default, while still allowing overrides for targeted assertions.

Method Description
_build_application Creates a HousingApplication with deterministic identity, financial, and review-state fields, then applies caller overrides.

Generated Test Shape

  • purchase_intent defaults to buy
  • connected_accounts defaults to an empty mapping
  • review_status defaults to pending
  • created_at is initialized to datetime.now()
  • All primary application fields are populated so scoring logic can run without fixture assembly in each test

backend/tests/usecase/test_credit_score_runtime.py

File path: backend/tests/usecase/test_credit_score_runtime.py

This helper builds a richer HousingApplication for runtime scoring tests. It includes Plaid-like connected accounts and a small transaction history so the runtime scorer can exercise affordability, liquidity, salary detection, and breakdown generation paths.

Method Description
_build_application Creates a HousingApplication with sample connected account data, balances, and transactions, then applies overrides.

Generated Test Shape

  • Includes one connected account under the plaid key
  • Includes sample checking balances
  • Includes salary and rent-style transactions
  • Sets archived=False and review_status="pending"
  • Supports contract-validation tests for Vertex AI responses

backend/tests/usecase/test_dummy_account_detection.py

File path: backend/tests/usecase/test_dummy_account_detection.py

This helper builds a HousingApplication used by dummy-account detection tests. It is tuned for heuristics that inspect repeated counterparties, balances, and account-holder name matching.

Method Description
_build_application Creates a HousingApplication with optional full_name and connected_accounts overrides for dummy-account detection scenarios.

Generated Test Shape

  • Defaults full_name to Tobias Andersen Yoshi
  • Uses created_at=datetime.now(timezone.utc)
  • Accepts caller-provided connected account structures
  • Keeps the rest of the housing application fields realistic for heuristic evaluation

backend/tests/ml/test_pipeline_validation.py

File path: backend/tests/ml/test_pipeline_validation.py

These helper constructors produce canonical ML inputs for pipeline validation. They are used to verify the gating rules around bureau availability, banking availability, coverage, and recency.

Method Description
_banking Creates a CanonicalOpenBankingData object with a provider name, account count, coverage days, and optional transaction set.
_bureau Creates a CanonicalBureauData object with score, recency, and optional fraud or AML flags.

Generated Validation Shapes

  • _banking defaults to a Plaid-like provider and one synthetic salary transaction
  • _bureau defaults to a TransUnion-like bureau score with a recent report date
  • Both helpers are suitable for testing stale-report and missing-data gates

backend/tests/interface/test_application_endpoint.py

File path: backend/tests/interface/test_application_endpoint.py

This builder creates a realistic application payload for endpoint testing. It mirrors the payload structure used by the application create flow, including connected accounts.

Method Description
build_payload Returns a dictionary that matches the application-create request shape used in endpoint tests.

Generated Payload Shape

  • email, phone, country, property type, purchase intent, budget range, savings, income, and currency
  • financing_consent, agree_policy, receive_updates
  • a connected_accounts object with a sample personal account entry

backend/tests/test_applicants.py

File path: backend/tests/test_applicants.py

These test harness helpers create isolated FastAPI and SQLAlchemy state for applicant endpoint tests. They are used to keep the consent-validation and applicant-management tests independent from the real backend database.

Method Description
_get_test_engine Builds a single in-memory SQLite engine and enables foreign keys on connect.
_create_session Drops and recreates all tables, then opens a fresh session for test isolation.
_create_customer Inserts an active customer with a generated API key and returns both the model and plaintext key.
_create_app Builds a FastAPI app, mounts the applicants router, registers the AuthError handler, and overrides get_db.

Test Harness Behavior

  • Uses SQLite in memory
  • Recreates schema per test to avoid cross-test contamination
  • Converts auth failures into 401 responses with WWW-Authenticate: Bearer
  • Supports tests that verify consent-grant validation behavior

Local Use vs Deployment Use

Asset Local Use Deployment Use
create_test_application.py Good for seeding and smoke testing against local backend instances Only when the target environment is intentionally being seeded
clear_housing_applications.py Good for resetting disposable databases Not appropriate unless data loss is intended
Test fixture builders Good for deterministic unit and integration tests Not used directly in deployment
FastAPI and SQLite test harness helpers Good for isolated test runs Not used directly in deployment

Error Handling

The scripts and helpers use simple, explicit failure handling.

  • create_test_application.py raises RuntimeError when the request fails, the backend returns an HTTP error, or the response body is not valid JSON.
  • clear_housing_applications.py aborts cleanly when confirmation is not received and always closes its session in a finally block.
  • The test fixture helpers are intentionally narrow and rely on Python object construction rather than runtime recovery.

Dependencies

Runtime and Library Dependencies

File Dependency Surface
backend/src/scripts/create_test_application.py argparse, json, os, urllib.request, urllib.error
backend/src/scripts/clear_housing_applications.py argparse, SessionLocal, SQLAlchemyHousingApplicationRepository
backend/tests/test_applicants.py FastAPI test client, SQLAlchemy engine/session helpers, AuthError handler
backend/tests/ml/test_pipeline_validation.py Canonical ML schema objects

Backend and Database Dependencies

  • Application-create and read helpers depend on the running backend service.
  • Database cleanup depends on the backend SQLAlchemy session configuration.
  • Test harness helpers depend on SQLite or the configured test database.

Testing Considerations

Scenario Why It Matters
Create helper with no APPLICATIONS_API_TOKEN Verifies the script can run against unauthenticated local backends when allowed
Create helper with invalid base URL Verifies transport failure handling
Create helper with non-JSON response Verifies the JSON decoding guard
Clear helper without confirmation Verifies the destructive path is blocked
Clear helper with --yes Verifies the non-interactive cleanup path
Fixture builders with override inputs Verifies that test payloads remain adjustable without rewriting the builders
Pipeline validation builders Verifies completeness, stale-report, and coverage gates with deterministic canonical data

Key Classes Reference

Class Responsibility
create_test_application.py Creates a test housing application through the backend API and prints a summary
clear_housing_applications.py Deletes all housing applications from the configured database
test_credit_scoring.py Builds a baseline HousingApplication test fixture
test_credit_score_runtime.py Builds a runtime scoring fixture with connected accounts and transactions
test_dummy_account_detection.py Builds a dummy-account detection fixture
test_pipeline_validation.py Builds canonical banking and bureau validation fixtures
test_application_endpoint.py Builds an application endpoint payload fixture
test_applicants.py Creates isolated FastAPI and SQLAlchemy test harnesses