How abhishekn1947/samgov-scraper works
Project type and purpose
• This is a Python-based ETL/data-pipeline script that automates contract data scraping from SAM.gov, processes and cleans the data, persists it to a PostgreSQL database on AWS RDS, exports CSV files, and delivers results via email through AWS SES.
• There is no separate frontend; it’s a single, scheduled or manually invoked backend workflow.
File structure
• Top level:
– main.py: the orchestration script.
– geckodriver: the Selenium driver executable.
– Final Csvs/Sample_data.csv: example output.
– Two PDF files with detailed documentation.
– README.md: project overview, dependencies, configuration, and process flow.
README for additional insights
• Key technologies: Selenium (with Firefox + geckodriver), Pandas, Boto3, psycopg2, python-dotenv.
• AWS services: SES for email, RDS PostgreSQL for persistence.
• Process flow steps 1–6 describe the end-to-end pipeline.
Diagramming recommendations
– Rectangles (blue) for internal modules