📄 SamGov.cs Documentation
IrbyImaging.UI.Services/SamGov.cs
A service that scrapes wage determination data from SAM.gov webpages using IronWebScraper.
📚 Table of Contents
- Overview & Purpose
- Dependencies & Technologies
- Data Models & View Models
- Key Functional Areas
- 4.1.
SamGovContainer Class - 4.2.
WageDeterminationData Model - 4.3.
WageDeterminationScraperScraper Class
- 4.1.
- Class Diagram (UML)
- Helper Methods & Utilities
- Error Handling & Logging
- Third‑Party Integrations
- Sample Usage & Code Snippets
- Conclusion
1. Overview & Purpose
The SamGov service provides a web scraping mechanism to retrieve Federal wage determination tables from SAM.gov (System for Award Management). It consists of:
- A data model (
WageDetermination) representing each wage entry. - A scraper (
WageDeterminationScraper) that navigates to a given search URL, parses the HTML table rows, and populates a collection ofWageDeterminationobjects.
This service can be integrated into imaging or procurement workflows, allowing automated updates of prevailing wage rates without manual intervention.
2. Dependencies & Technologies
| Dependency | Purpose |
|---|---|
IronWebScraper |
Core web‑scraping framework (CSS selectors). |
Newtonsoft.Json |
JSON serialization/deserialization (unused). |
System.Net.Http |
HTTP requests (via base scraper). |
Azure |
Azure SDK reference (currently unused). |
.NET Standard |
Built on C# / .NET for cross‑platform support. |
ℹ️ Note: Ensure that your project includes the NuGet packages:
IronWebScraperNewtonsoft.Json
3. Data Models & View Models
3.1 WageDetermination
A simple POCO (Plain Old CLR Object) that holds:
| Property | Type | Description |
|---|---|---|
WageTitle |
string | Title or classification of the wage group. |
MinimumWage |
string | The base hourly wage. |
FringeBenefits |
string | The fringe benefit rate for the group. |
public class WageDetermination
{
public string WageTitle { get; set; }
public string MinimumWage { get; set; }
public string FringeBenefits { get; set; }
}
4. Key Functional Areas
4.1 SamGov Container Class
- Namespace:
IrbyImaging.UI.Services - Purpose: Acts as a logical container for related types (
WageDetermination,WageDeterminationScraper). - Public Interface: None directly; all functionality is exposed by nested types.
4.2 WageDetermination Data Model
- Describes the structure of a single row in the wage determination table.
- Used to hold parsed data from the HTML page.
4.3 WageDeterminationScraper Scraper Class
Inherits from IronWebScraper.WebScraper and overrides two lifecycle methods:
| Member | Signature | Description |
|---|---|---|
| Constructor | WageDeterminationScraper(string searchUrl) |
Initializes with the target SAM.gov search URL. |
| Init() | public override void Init() |
Entry point: configures logging and fires the initial request. |
| Parse(Response) | public override void Parse(Response response) |
Parses each table row (<tr>) and extracts <td> cell text. |
| WageData (property) | List<WageDetermination> WageData { get; } |
Accumulates all parsed wage determinations. |
public class WageDeterminationScraper : WebScraper
{
private readonly string _searchUrl;
public List<WageDetermination> WageData { get; private set; }
public WageDeterminationScraper(string searchUrl)
{
_searchUrl = searchUrl;
WageData = new List<WageDetermination>();
}
public override void Init()
{
LoggingLevel = LogLevel.None;
Request(_searchUrl, Parse);
}
public override void Parse(IronWebScraper.Response response)
{
var rows = response.Css("table tr");
foreach (var row in rows)
{
var cells = row.Css("td");
if (cells.Length >= 3)
{
WageData.Add(new WageDetermination
{
WageTitle = cells[0].TextContentClean,
MinimumWage = cells[1].TextContentClean,
FringeBenefits = cells[2].TextContentClean
});
}
}
}
}
5. Class Diagram (UML)
classDiagram
class SamGov {
<<namespace>>
}
SamGov <|-- WageDetermination
SamGov <|-- WageDeterminationScraper
class WageDetermination {
+string WageTitle
+string MinimumWage
+string FringeBenefits
}
class WageDeterminationScraper {
-string _searchUrl
+List~WageDetermination~ WageData
+WageDeterminationScraper(string searchUrl)
+override void Init()
+override void Parse(Response response)
}
6. Helper Methods & Utilities
Response.Css(selector)
A method from IronWebScraper that queries HTML nodes using CSS selectors.TextContentClean
Strips whitespace and control characters fromInnerText.
🔧 No additional custom utilities are defined in this file.
7. Error Handling & Logging
- LoggingLevel: Set to
LogLevel.Noneby default; adjust for debugging. - Error Handling: Inherits try/catch and retry policies from
WebScraperbase class. - Recommendations:
- Wrap
Start()/StartAsync()calls intry/catchto handle network or parsing errors. - Validate that the returned HTML structure matches the expected
<table>format.
- Wrap
8. Third‑Party Integrations
| Library | Usage |
|---|---|
| IronWebScraper | Core scraping engine & CSS selection. |
| Newtonsoft.Json | (Imported) JSON serialization. |
| Azure SDK | (Imported) Potential future use. |
9. Sample Usage & Code Snippets
9.1 Instantiating & Running the Scraper
using IrbyImaging.UI.Services;
using System;
using System.Threading.Tasks;
class Program
{
static async Task Main(string[] args)
{
// 1. Define the SAM.gov search URL for wage determinations
string samGovUrl = "https://sam.gov/wage-determinations/...";
// 2. Create the scraper
var scraper = new SamGov.WageDeterminationScraper(samGovUrl);
// 3. Start scraping (async)
await scraper.StartAsync();
// 4. Retrieve results
var wages = scraper.WageData;
// 5. Display/output
foreach (var wd in wages)
{
Console.WriteLine($"{wd.WageTitle}: {wd.MinimumWage} (+{wd.FringeBenefits})");
}
}
}
Notes
- Use
Start()orStartAsync()depending on your hosting environment.- Ensure network permissions to access SAM.gov.
9.2 Filtering & Serialization
// Filter wages above a threshold
var highWages = wages
.Where(w => Decimal.Parse(w.MinimumWage.Trim('$')) > 25.00m)
.ToList();
// Serialize to JSON
string json = JsonConvert.SerializeObject(highWages, Formatting.Indented);
Console.WriteLine(json);
10. Conclusion
The SamGov service in SamGov.cs provides a lightweight, extensible solution to programmatically ingest wage determination data from SAM.gov. By leveraging IronWebScraper, it abstracts HTTP requests and HTML parsing, exposing a simple API:
- Initialize with a SAM.gov URL.
- Run the scraper (
Init➔Parse). - Consume the strongly‑typed
WageDeterminationlist.
🚀 Next Steps:
- Enhance error resilience (retries, timeouts).
- Support pagination or multiple tables.
- Add unit tests for parsing logic.
- Integrate into data pipelines or Azure Functions for periodic updates.
Happy scraping! 🕸️🧹