# abhishekn1947/samgov-scraper architecture

> Architecture diagram of the GitHub repository abhishekn1947/samgov-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/abhishekn1947/samgov-scraper
- Repository: https://github.com/abhishekn1947/samgov-scraper

## Overview

##### Project type and purpose
- This is a Python-based ETL/data-pipeline script that automates contract data scraping from SAM.gov, processes and cleans the data, persists it to a PostgreSQL database on AWS RDS, exports CSV files, and delivers results via email through AWS SES.
- There is no separate frontend; it’s a single, scheduled or manually invoked backend workflow.

##### File structure
- Top level:
– main.py: the orchestration script.
– geckodriver: the Selenium driver executable.
– Final Csvs/Sample_data.csv: example output.
– Two PDF files with detailed documentation.
– README.md: project overview, dependencies, configuration, and process flow.

##### README for additional insights
- Key technologies: Selenium (with Firefox + geckodriver), Pandas, Boto3, psycopg2, python-dotenv.
- AWS services: SES for email, RDS PostgreSQL for persistence.
- Process flow steps 1–6 describe the end-to-end pipeline.

##### Diagramming recommendations
– Rectangles (blue) for internal modules

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% ETL Pipeline Flow (Vertical)
    Scheduler["Scheduler/Trigger"]:::internal
    ConfigLoader["Configuration Loader\n(.env via python-dotenv)"]:::internal
    Scraper["Scraper Module\n(Selenium)"]:::internal
    Geckodriver["geckodriver"]:::file
    SAM["SAM.gov\n(External Website)"]:::external
    Processor["Data Processor\n(Pandas)"]:::internal
    Storage["AWS RDS PostgreSQL"]:::db
    FileExport["CSV Export\n(Final Csvs/)"]:::file
    SES["AWS SES\n(Email Service)"]:::awsCloud
    Recipients["Email Recipients"]:::email

    Scheduler -->|triggers| ConfigLoader
    ConfigLoader -->|loads config for| Scraper
    Scraper -->|uses| Geckodriver
    Scraper -->|scrapes data from| SAM
    Scraper -->|passes raw data to| Processor
    Processor -->|inserts cleaned data into| Storage
    Processor -->|exports results to| FileExport
    FileExport -->|attaches CSVs and calls| SES
    SES -->|sends emails to| Recipients

    %% Click Events
    click Scheduler "https://github.com/abhishekn1947/samgov-scraper/blob/main/README.md"
    click ConfigLoader "https://github.com/abhishekn1947/samgov-scraper/blob/main/main.py"
    click Scraper "https://github.com/abhishekn1947/samgov-scraper/blob/main/main.py"
    click Geckodriver "https://github.com/abhishekn1947/samgov-scraper/tree/main/geckodriver"
    click Processor "https://github.com/abhishekn1947/samgov-scraper/blob/main/main.py"
    click Storage "https://github.com/abhishekn1947/samgov-scraper/blob/main/main.py"
    click FileExport "https://github.com/abhishekn1947/samgov-scraper/tree/main/Final Csvs/"
    click SES "https://github.com/abhishekn1947/samgov-scraper/blob/main/main.py"

    %% Styles
    classDef internal fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef db fill:#90EE90,stroke:#000,stroke-width:1px
    classDef awsCloud fill:#FFA500,stroke:#000,stroke-width:1px
    classDef file fill:#D3D3D3,stroke:#000,stroke-width:1px
    classDef email fill:#FFFFE0,stroke:#000,stroke-width:1px
    classDef external fill:#F0F0F0,stroke:#000,stroke-width:1px
```
