# shayansm2/tech-career-explorer architecture

> Architecture diagram of the GitHub repository shayansm2/tech-career-explorer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shayansm2/tech-career-explorer
- Repository: https://github.com/shayansm2/tech-career-explorer

## Overview

##### Main Components
- Data Ingestion (Crawlers & Data Loaders):
– Crawler modules (found under src/crawlers and google-scripts) are responsible for scanning listing and detail pages.
– Data loaders (see src/data_loaders) kick off crawling, invoking crawlers to retrieve raw job data.
- Data Transformation:
– DBT components (located under src/dbt/job_positions) execute transformation on the raw data to create final tables (fact_positions and fact_metadata). Also include pipelines (src/pipelines) that coordinate DBT runs and other transformation processes.
- Data Exporting:
– Data exporter modules (under src/data_exporters) take the cleaned and transformed data to load them into the Data Warehouse.
- Data Warehouse:
– PostgreSQL serves as the central repository for production (with the backup in DuckDB). Indicate the two main tables (fact_metadata and fact_positions) along with techniques like partitioning and clustering.
- Workflow Orchestration:
– Mage-AI is used to manage ETL jobs; include its integration and how it coordinates batch processing.
- Dashboard:
– Metabase connects to the warehouse and provides visualization/analytics.
- Additional Components:
– Configuration files and YAML files (src/configs, io_config.yaml, metadata files) guide execution and transformation rules.
– Custom modules, extensions, and schema definitions add to custom logic and data definitions.

##### Relationships and Interactions
– From external web sources (LinkedIn, Glassdoor, Relocate) to the Crawlers.
– From Crawlers to Data Loaders and then to Data Exporters performing minor cleaning and formatting tasks.
– From Data Exporters to the Data Warehouse (PostgreSQL) with a “Backup/Dev” branch to DuckDB.
– The orchestration platform (Mage-AI) should be centralized showing how it triggers and monitors ingestion and transformation pipelines.

##### Architectural Patterns and Design Principles
- The architecture combines an ELT approach with selective ETL processes:
– Crawlers extract and initially transform data before loading.
– DBT handles heavy transformation, following batch processing best practices.
- Separation of concern is evident in distinct directories for crawlers, loaders, exporters, pipelines, and transformations.
- Modularity: Each component (crawlers, pipelines, dbt projects) is self-contained for scalability and maintainability.
- Use of configuration files and a clear modular structure supports extensibility.

##### Detail and Clarity
- Be very detailed in subdividing the modules especially given the rich file structure. For instance, highlight that the DBT process includes separate folders for staging and core models.

## Diagram (Mermaid)

```mermaid
graph TD
    subgraph "Configurations"
        configs("Configs"):::config
        ioConfig("IO Config"):::config
        metadataConfig("Metadata Config"):::config
        customModules("Custom Modules"):::config
    end

    subgraph "Data Ingestion Layer"
        LN("LinkedIn Source"):::external
        glassdoor("Glassdoor Source"):::external
        relocate("Relocate Source"):::external

        crawlers("Crawlers"):::ingestion
        googleScripts("Google Scripts"):::ingestion
        dataLoaders("Data Loaders"):::ingestion
    end

    subgraph "Data Transformation Layer"
        dbt("DBT Transformation"):::transformation
        pipelines("Pipelines"):::transformation
    end

    subgraph "Data Export & Storage Layer"
        dataExporters("Data Exporters"):::ingestion
        dataWarehouse("Data Warehouse (PostgreSQL)"):::storage
        duckDB("DuckDB Backup"):::storage
    end

    mageAI("Mage-AI Orchestration"):::orchestration
    dashboard("Dashboard (Metabase)"):::dashboard

    %% Data extraction from external sources
    LN -->|"JobDataExtraction"| crawlers
    glassdoor -->|"JobDataExtraction"| crawlers
    relocate -->|"JobDataExtraction"| crawlers

    %% Ingestion flow
    crawlers -->|"InvokeCrawling"| dataLoaders
    googleScripts -->|"ScriptExtraction"| dataLoaders

    %% Data loading and exporting
    dataLoaders -->|"CleanFormatData"| dataExporters
    dataExporters -->|"LoadData"| dataWarehouse

    %% Data transformation and backup
    dataWarehouse -->|"RawData"| dbt
    dbt -->|"TransformedData_(fact_metadata, fact_positions)"| dataWarehouse
    dataWarehouse -->|"DataBackup"| duckDB

    %% Data provisioning for dashboard
    dataWarehouse -->|"ProvideData"| dashboard

    %% Workflow orchestration triggers
    mageAI -->|"TriggerJobs"| dataLoaders
    mageAI -->|"TriggerTransformation"| dbt
    mageAI -->|"TriggerJobs"| dataExporters

    %% Configuration connections
    configs -->|"CrawlConfigs"| crawlers
    ioConfig -->|"LoaderConfig"| dataLoaders
    metadataConfig -->|"MetadataConfig"| dbt
    customModules -->|"CustomModules"| pipelines

    %% Pipeline coordination for DBT
    pipelines -->|"CoordinateDBTJobs"| dbt

    %% Click Events
    click crawlers "https://github.com/shayansm2/tech-career-explorer/tree/main/src/crawlers"
    click googleScripts "https://github.com/shayansm2/tech-career-explorer/tree/main/google-scripts"
    click dataLoaders "https://github.com/shayansm2/tech-career-explorer/tree/main/src/data_loaders"
    click dbt "https://github.com/shayansm2/tech-career-explorer/tree/main/src/dbt/job_positions"
    click pipelines "https://github.com/shayansm2/tech-career-explorer/tree/main/src/pipelines"
    click dataExporters "https://github.com/shayansm2/tech-career-explorer/tree/main/src/data_exporters"
    click mageAI "https://github.com/shayansm2/tech-career-explorer/blob/main/mageai.Dockerfile"
    click dashboard "https://github.com/shayansm2/tech-career-explorer/tree/main/metabase-data"
    click configs "https://github.com/shayansm2/tech-career-explorer/tree/main/src/configs"
    click ioConfig "https://github.com/shayansm2/tech-career-explorer/blob/main/src/io_config.yaml"
    click metadataConfig "https://github.com/shayansm2/tech-career-explorer/blob/main/src/metadata.yaml"
    click customModules "https://github.com/shayansm2/tech-career-explorer/tree/main/src/custom"

    %% Styles
    classDef ingestion fill:#a3d8f4,stroke:#000,stroke-width:1px;
    classDef transformation fill:#d4f4a3,stroke:#000,stroke-width:1px;
    classDef storage fill:#f4d8a3,stroke:#000,stroke-width:1px;
    classDef orchestration fill:#d8a3f4,stroke:#000,stroke-width:1px;
    classDef dashboard fill:#f4a3d8,stroke:#000,stroke-width:1px;
    classDef config fill:#e0e0e0,stroke:#000,stroke-width:1px;
    classDef external fill:#ffe6a3,stroke:#000,stroke-width:1px;
```
