# xenoswarlocks/super-duper-engine architecture

> Architecture diagram of the GitHub repository xenoswarlocks/super-duper-engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xenoswarlocks/super-duper-engine
- Repository: https://github.com/xenoswarlocks/super-duper-engine

## Overview

##### Core Project Type & Purpose
- Project Type: A Python-based automated web scraper.
- Purpose: To scrape lead generation job listings from Naukri.com by executing a three-step workflow consisting of initialization/search, data extraction, and data cleaning/processing.
- Key Indicators: The README details a scraping pipeline with modules named firstStep.py, secondStep.py, and thirdStep.py, along with CSV output files generated in the output folder.

##### File Structure
- Top-Level Files:
- LICENSE and README.md (project documentation and legal info).
- __main__.py: Acts as the main entry point to kick off the scraping process.
- Backend Directory:
- Contains three modules:
▸ firstStep.py: Initializes the Selenium WebDriver, performs page navigation, and initiates the search on Naukri.com.
▸ secondStep.py: Extracts job details from multiple pages and collects raw data.
▸ thirdStep.py: Cleans and processes the collected data (e.g., removing outdated entries, de-duplication).
- Output Directory:
- Holds CSV files generated during the process (raw data and cleaned data).

a. Main Execution Flow:
- __main__.py: The central controller that orchestrates the overall pipeline.
b. Backend Components:
- FirstStep Module: Handles browser initialization (using Selenium + ChromeDriver), Naukri.com navigation, and search execution.
- SecondStep Module: Responsible for the extraction of data from the results, handling pagination (batch processing up to 20 jobs), and writing raw CSV output.
- ThirdStep Module: Reads raw CSV data, applies data cleaning (removing old or duplicate entries), and writes cleaned data to a separate CSV.
c. External Dependencies & Data Sources:
- Naukri.com: The external source for lead generation job listings.
- Chrome Browser + ChromeDriver: For interacting with the website.
- Python Libraries: Selenium for web automation, Pandas and NumPy for data processing.
d. Data Persistence:
- CSV Files: Two files in the output folder (companies_data.csv for raw data, companies_data_cleaned.csv for cleaned data).

##### Relationships and Interactions
- The execution flow starts in __main__.py, which sequentially calls the backend modules.
- The firstStep module sets up the automated browser session and initiates the targeted search.
- Upon successful search execution, the secondStep module is invoked to extract data (with built-in smart waits and error handling).
- Finally, the thirdStep module processes the extracted data, handling deduplication and exporting the final results.
- Data flows from web pages (external Naukri.com) through the Selenium-controlled browser and then into CSV storage.
- The modules interact sequentially, and error handling exists to manage network issues, timeouts, or data processing errors.

##### Architectural Patterns and Design Principles
- Modular Pipeline: The project is segmented into clearly defined modules (steps) that handle specific responsibilities.
- Separation of Concerns: Each backend module has a distinct purpose—initialization/navigation, data extraction, and data cleaning.
- Resumable & Batch Processing: The design emphasizes batch processing and resume capability to ensure robust operation and error recovery.
- External Dependency Management: Reliance on third-party libraries (Selenium, Pandas, NumPy) and integration with external services (Chrome, Naukri.com).

## Diagram (Mermaid)

```mermaid
graph TD
  %% Execution Pipeline
  subgraph "Execution Pipeline"
    MainController("Main Controller (__main__.py)"):::controller
    Step1("FirstStep Module (Browser Initialization and Navigation)"):::execution
    Step2("SecondStep Module (Data Extraction from Naukri.com)"):::execution
    Step3("ThirdStep Module (Data Cleaning & Processing)"):::execution
  end

  %% Data Persistence
  subgraph "Data Persistence"
    RawCSV(("Raw CSV Output")):::data
    CleanCSV(("Cleaned CSV Output")):::data
  end

  %% External Dependencies
  subgraph "External Dependencies"
    Naukri("Naukri.com"):::external
    Chrome("Chrome Browser/ChromeDriver"):::external
  end

  %% Python Libraries
  subgraph "Python Libraries"
    Selenium("Selenium"):::libraries
    Pandas("Pandas"):::libraries
    NumPy("NumPy"):::libraries
  end

  %% Main Execution Flow
  MainController -->|"triggers"| Step1
  Step1 -->|"navigatesTo"| Step2
  Step2 -->|"extractsData"| Step3
  Step2 -->|"writesRawCSV"| RawCSV
  Step3 -->|"writesCleanedCSV"| CleanCSV
  Step3 -->|"errorHandling"| Step1

  %% External Interactions
  Naukri -->|"providesContent"| Step1
  Chrome -->|"controlsBrowser"| Step1

  %% Library Dependencies
  Selenium -->|"automation"| Step1
  Pandas -->|"dataProcessing"| Step3
  NumPy -->|"dataProcessing"| Step3

  %% Click Events
  click MainController "https://github.com/xenoswarlocks/super-duper-engine/blob/main/__main__.py"
  click Step1 "https://github.com/xenoswarlocks/super-duper-engine/blob/main/backend/firstStep.py"
  click Step2 "https://github.com/xenoswarlocks/super-duper-engine/blob/main/backend/secondStep.py"
  click Step3 "https://github.com/xenoswarlocks/super-duper-engine/blob/main/backend/thirdStep.py"
  click RawCSV "https://github.com/xenoswarlocks/super-duper-engine/blob/main/output/companies_data.csv"
  click CleanCSV "https://github.com/xenoswarlocks/super-duper-engine/blob/main/output/companies_data_cleaned.csv"

  %% Styles
  classDef controller fill:#f9d,stroke:#333,stroke-width:2px;
  classDef execution fill:#add8e6,stroke:#333,stroke-width:1px;
  classDef external fill:#ffcccb,stroke:#333,stroke-width:1px;
  classDef data fill:#90ee90,stroke:#333,stroke-width:1px;
  classDef libraries fill:#ffffcc,stroke:#333,stroke-width:1px;
```
