📄 SamGov.cs Documentation

IrbyImaging.UI.Services/SamGov.cs
A service that scrapes wage determination data from SAM.gov webpages using IronWebScraper.


📚 Table of Contents

  1. Overview & Purpose
  2. Dependencies & Technologies
  3. Data Models & View Models
  4. Key Functional Areas
    • 4.1. SamGov Container Class
    • 4.2. WageDetermination Data Model
    • 4.3. WageDeterminationScraper Scraper Class
  5. Class Diagram (UML)
  6. Helper Methods & Utilities
  7. Error Handling & Logging
  8. Third‑Party Integrations
  9. Sample Usage & Code Snippets
  10. Conclusion

1. Overview & Purpose

The SamGov service provides a web scraping mechanism to retrieve Federal wage determination tables from SAM.gov (System for Award Management). It consists of:

  • A data model (WageDetermination) representing each wage entry.
  • A scraper (WageDeterminationScraper) that navigates to a given search URL, parses the HTML table rows, and populates a collection of WageDetermination objects.

This service can be integrated into imaging or procurement workflows, allowing automated updates of prevailing wage rates without manual intervention.


2. Dependencies & Technologies

Dependency Purpose
IronWebScraper Core web‑scraping framework (CSS selectors).
Newtonsoft.Json JSON serialization/deserialization (unused).
System.Net.Http HTTP requests (via base scraper).
Azure Azure SDK reference (currently unused).
.NET Standard Built on C# / .NET for cross‑platform support.

ℹ️ Note: Ensure that your project includes the NuGet packages:

  • IronWebScraper
  • Newtonsoft.Json

3. Data Models & View Models

3.1 WageDetermination

A simple POCO (Plain Old CLR Object) that holds:

Property Type Description
WageTitle string Title or classification of the wage group.
MinimumWage string The base hourly wage.
FringeBenefits string The fringe benefit rate for the group.
public class WageDetermination
{
    public string WageTitle { get; set; }
    public string MinimumWage { get; set; }
    public string FringeBenefits { get; set; }
}

4. Key Functional Areas

4.1 SamGov Container Class

  • Namespace: IrbyImaging.UI.Services
  • Purpose: Acts as a logical container for related types (WageDetermination, WageDeterminationScraper).
  • Public Interface: None directly; all functionality is exposed by nested types.

4.2 WageDetermination Data Model

  • Describes the structure of a single row in the wage determination table.
  • Used to hold parsed data from the HTML page.

4.3 WageDeterminationScraper Scraper Class

Inherits from IronWebScraper.WebScraper and overrides two lifecycle methods:

Member Signature Description
Constructor WageDeterminationScraper(string searchUrl) Initializes with the target SAM.gov search URL.
Init() public override void Init() Entry point: configures logging and fires the initial request.
Parse(Response) public override void Parse(Response response) Parses each table row (<tr>) and extracts <td> cell text.
WageData (property) List<WageDetermination> WageData { get; } Accumulates all parsed wage determinations.
public class WageDeterminationScraper : WebScraper
{
    private readonly string _searchUrl;
    public List<WageDetermination> WageData { get; private set; }

    public WageDeterminationScraper(string searchUrl)
    {
        _searchUrl = searchUrl;
        WageData = new List<WageDetermination>();
    }

    public override void Init()
    {
        LoggingLevel = LogLevel.None;
        Request(_searchUrl, Parse);
    }

    public override void Parse(IronWebScraper.Response response)
    {
        var rows = response.Css("table tr");
        foreach (var row in rows)
        {
            var cells = row.Css("td");
            if (cells.Length >= 3)
            {
                WageData.Add(new WageDetermination
                {
                    WageTitle       = cells[0].TextContentClean,
                    MinimumWage     = cells[1].TextContentClean,
                    FringeBenefits  = cells[2].TextContentClean
                });
            }
        }
    }
}

5. Class Diagram (UML)

classDiagram
    class SamGov {
        <<namespace>>
    }

    SamGov <|-- WageDetermination
    SamGov <|-- WageDeterminationScraper

    class WageDetermination {
      +string WageTitle
      +string MinimumWage
      +string FringeBenefits
    }

    class WageDeterminationScraper {
      -string _searchUrl
      +List~WageDetermination~ WageData
      +WageDeterminationScraper(string searchUrl)
      +override void Init()
      +override void Parse(Response response)
    }

6. Helper Methods & Utilities

  • Response.Css(selector)
    A method from IronWebScraper that queries HTML nodes using CSS selectors.
  • TextContentClean
    Strips whitespace and control characters from InnerText.

🔧 No additional custom utilities are defined in this file.


7. Error Handling & Logging

  • LoggingLevel: Set to LogLevel.None by default; adjust for debugging.
  • Error Handling: Inherits try/catch and retry policies from WebScraper base class.
  • Recommendations:
    • Wrap Start() / StartAsync() calls in try/catch to handle network or parsing errors.
    • Validate that the returned HTML structure matches the expected <table> format.

8. Third‑Party Integrations

Library Usage
IronWebScraper Core scraping engine & CSS selection.
Newtonsoft.Json (Imported) JSON serialization.
Azure SDK (Imported) Potential future use.

9. Sample Usage & Code Snippets

9.1 Instantiating & Running the Scraper

using IrbyImaging.UI.Services;
using System;
using System.Threading.Tasks;

class Program
{
    static async Task Main(string[] args)
    {
        // 1. Define the SAM.gov search URL for wage determinations
        string samGovUrl = "https://sam.gov/wage-determinations/...";

        // 2. Create the scraper
        var scraper = new SamGov.WageDeterminationScraper(samGovUrl);

        // 3. Start scraping (async)
        await scraper.StartAsync(); 

        // 4. Retrieve results
        var wages = scraper.WageData;

        // 5. Display/output
        foreach (var wd in wages)
        {
            Console.WriteLine($"{wd.WageTitle}: {wd.MinimumWage} (+{wd.FringeBenefits})");
        }
    }
}

Notes

  • Use Start() or StartAsync() depending on your hosting environment.
  • Ensure network permissions to access SAM.gov.

9.2 Filtering & Serialization

// Filter wages above a threshold
var highWages = wages
    .Where(w => Decimal.Parse(w.MinimumWage.Trim('$')) > 25.00m)
    .ToList();

// Serialize to JSON
string json = JsonConvert.SerializeObject(highWages, Formatting.Indented);
Console.WriteLine(json);

10. Conclusion

The SamGov service in SamGov.cs provides a lightweight, extensible solution to programmatically ingest wage determination data from SAM.gov. By leveraging IronWebScraper, it abstracts HTTP requests and HTML parsing, exposing a simple API:

  1. Initialize with a SAM.gov URL.
  2. Run the scraper (Init ➔ Parse).
  3. Consume the strongly‑typed WageDetermination list.

🚀 Next Steps:

  • Enhance error resilience (retries, timeouts).
  • Support pagination or multiple tables.
  • Add unit tests for parsing logic.
  • Integrate into data pipelines or Azure Functions for periodic updates.

Happy scraping! 🕸️🧹