# rosdatatech/rdt architecture

> Architecture diagram of the GitHub repository rosdatatech/rdt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rosdatatech/rdt
- Repository: https://github.com/rosdatatech/rdt

## Overview

##### Project type and purpose
- This is a research‑oriented data‑analysis and reporting tool rather than a classic web full‑stack app.
- It ingests curricula/program data from university websites (via Parsers), pulls job postings (rdt_hhru), stores everything in PostgreSQL, runs ML/NLP analysis (rdt_ml), and produces interactive views (PyQt5 in rdt_db_viewer) and PDF reports (rdt_pdf).

##### File structure
- Top‑level “Parsers/” folder contains domain‑specific ETL scripts. Each subfolder (Geology, IT, etc.) has …parser.py (scrapes/parses HTML), db_uploader.py (pushes parsed records into the DB).
- “main/” holds the core orchestration:
- main.py – entry point coordinating the pipeline
- rdt_database.py – DB connectivity and schema setup
- rdt_hhru.py – job‑listing scraper
- rdt_ml.py – analysis pipeline (using scikit‑learn, sentence‑transformers, gensim, numpy)
- rdt_pdf.py – report generation
- rdt_db_viewer.py – PyQt5 GUI for exploring stored data
- requirements.txt lists all dependencies; README describes the tech stack and high‑level flow.

##### Core components
a. Data Sources (external)
- University websites (various domains)
- Job portal (hh.ru or equivalent)
b. Ingestion & ETL Layer (“Parsers/” + rdt_hhru)
- Parsing scripts and download_func modules
- db_uploader scripts
c. Storage Layer
- rdt_database module for ORM/connection
d. Processing/Analytics Layer
- rdt_ml module — NLP embedding (sentence‑transformers), topic modeling (gensim), classification/regression (scikit‑learn)
e. Presentation & Reporting Layer
- rdt_db_viewer (PyQt5 GUI)
- rdt_pdf (PDF report generator)
f. Orchestrator
- main.py – sequences ingestion, analysis, and reporting

##### Relationships & Interactions
- Data flow from external sources → Parsers & rdt_hhru → PostgreSQL
- rdt_database sits between PostgreSQL and all internal modules needing data
- rdt_ml reads from DB, writes analysis results back to DB or to a report object
- rdt_db_viewer reads from DB to display interactive dashboards
- rdt_pdf reads from DB and rdt_ml outputs to generate final PDF
- main.py invokes each component in sequence (ingest → analyze → present)

##### Architectural patterns & principles
- Modular ETL: one submodule per domain, each with clear responsibility (parse vs. upload)
- Layered architecture: Ingestion, Storage, Processing, Presentation
- Single orchestrator entry point to simplify end‑to‑end runs
- Separation of concerns: database connectivity isolated in rdt_database

##### Final advice
- Be exhaustive in mapping every script/module to one of the layers.
- Don’t overcomplicate: it’s better to show many small components than to hide them in a monolith.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Sources
    subgraph "External Data Sources"
        ext1["University Websites"]:::external
        ext2["Job Portal (hh.ru)"]:::external
    end

    %% Ingestion & ETL Layer
    subgraph "Ingestion & ETL Layer"
        %% Geology Domain Parsers
        subgraph "Geology Parsers"
            MP_MGRI["MGRIparser.py"]:::etl
            MU_MGRI["db_uploader.py"]:::etl
            MP_MGY["MGYparser.py"]:::etl
            MU_MGY["db_uploader.py"]:::etl
            MP_MIIG["MIIGAIKparser.py"]:::etl
            MU_MIIG["db_uploader.py"]:::etl
            MP_RGY["RGYparser.py"]:::etl
            MU_RGY["db_uploader.py"]:::etl
            MU_OTHER["db_uploader_other.py"]:::etl
        end
        %% IT Domain Parsers
        ITGroup["IT Parsers"]:::etl
        %% Other Domains
        JurisGroup["Jurisprudence Parsers"]:::etl
        MedGroup["Medicine Parsers"]:::etl
        PedGroup["Pedagogy Parsers"]:::etl
        %% Job Scraper
        JobScraper["rdt_hhru.py"]:::etl
    end

    %% Storage Layer
    subgraph "Storage Layer"
        DBMod["rdt_database.py"]:::storage
        PostgreSQL[(PostgreSQL)]:::storage
    end

    %% Processing & Analytics Layer
    subgraph "Processing & Analytics Layer"
        ML["rdt_ml.py"]:::analytics
    end

    %% Presentation & Reporting Layer
    subgraph "Presentation & Reporting Layer"
        GUI["rdt_db_viewer.py"]:::presentation
        PDF["rdt_pdf.py"]:::presentation
    end

    %% Orchestrator
    Orchestrator["main.py"]:::orchestrator

    %% Data Flows
    ext1 -->|ingest curricula| MP_MGRI
    ext1 --> MP_MGY
    ext1 --> MP_MIIG
    ext1 --> MP_RGY
    ext1 --> MU_OTHER
    ext1 --> ITGroup
    ext1 --> JurisGroup
    ext1 --> MedGroup
    ext1 --> PedGroup
    ext2 -->|scrape jobs| JobScraper

    %% Parsers to DB
    MP_MGRI --> DBMod
    MU_MGRI --> DBMod
    MP_MGY --> DBMod
    MU_MGY --> DBMod
    MP_MIIG --> DBMod
    MU_MIIG --> DBMod
    MP_RGY --> DBMod
    MU_RGY --> DBMod
    MU_OTHER --> DBMod
    ITGroup --> DBMod
    JurisGroup --> DBMod
    MedGroup --> DBMod
    PedGroup --> DBMod
    JobScraper --> DBMod

    %% Storage interactions
    DBMod --> PostgreSQL
    PostgreSQL --> ML
    ML --> PostgreSQL
    PostgreSQL --> GUI
    PostgreSQL --> PDF

    %% Orchestration
    Orchestrator --> MP_MGRI
    Orchestrator --> MP_MGY
    Orchestrator --> MP_MIIG
    Orchestrator --> MP_RGY
    Orchestrator --> MU_OTHER
    Orchestrator --> ITGroup
    Orchestrator --> JurisGroup
    Orchestrator --> MedGroup
    Orchestrator --> PedGroup
    Orchestrator --> JobScraper
    Orchestrator --> ML
    Orchestrator --> GUI
    Orchestrator --> PDF

    %% Click Events
    click MP_MGRI "https://github.com/rosdatatech/rdt/blob/main/Parsers/Geology/MGRI/MGRIparser.py"
    click MU_MGRI "https://github.com/rosdatatech/rdt/blob/main/Parsers/Geology/MGRI/db_uploader.py"
    click MP_MGY "https://github.com/rosdatatech/rdt/blob/main/Parsers/Geology/MGY/MGYparser.py"
    click MU_MGY "https://github.com/rosdatatech/rdt/blob/main/Parsers/Geology/MGY/db_uploader.py"
    click MP_MIIG "https://github.com/rosdatatech/rdt/blob/main/Parsers/Geology/MIIGAiK/MIIGAIKparser.py"
    click MU_MIIG "https://github.com/rosdatatech/rdt/blob/main/Parsers/Geology/MIIGAiK/db_uploader.py"
    click MP_RGY "https://github.com/rosdatatech/rdt/blob/main/Parsers/Geology/RGY/RGYparser.py"
    click MU_RGY "https://github.com/rosdatatech/rdt/blob/main/Parsers/Geology/RGY/db_uploader.py"
    click MU_OTHER "https://github.com/rosdatatech/rdt/blob/main/Parsers/Geology/db_uploader_other.py"
    click ITGroup "https://github.com/rosdatatech/rdt/tree/main/Parsers/IT"
    click JurisGroup "https://github.com/rosdatatech/rdt/tree/main/Parsers/Jurisprudence"
    click MedGroup "https://github.com/rosdatatech/rdt/blob/main/Parsers/Medicine/db_uploader.py"
    click PedGroup "https://github.com/rosdatatech/rdt/tree/main/Parsers/Pedagogy"
    click JobScraper "https://github.com/rosdatatech/rdt/blob/main/main/rdt_hhru.py"
    click DBMod "https://github.com/rosdatatech/rdt/blob/main/main/rdt_database.py"
    click ML "https://github.com/rosdatatech/rdt/blob/main/main/rdt_ml.py"
    click GUI "https://github.com/rosdatatech/rdt/blob/main/main/rdt_db_viewer.py"
    click PDF "https://github.com/rosdatatech/rdt/blob/main/main/rdt_pdf.py"
    click Orchestrator "https://github.com/rosdatatech/rdt/blob/main/main/main.py"

    %% Styles
    classDef etl fill:#ADD8E6,stroke:#000,stroke-width:1px;
    classDef storage fill:#98FB98,stroke:#000,stroke-width:1px;
    classDef analytics fill:#FFA500,stroke:#000,stroke-width:1px;
    classDef presentation fill:#DDA0DD,stroke:#000,stroke-width:1px;
    classDef orchestrator fill:#FFC0CB,stroke:#000,stroke-width:2px;
    classDef external fill:#F5F5F5,stroke:#000,stroke-dasharray: 5 5;
    class Orchestrator orchestrator
    class ext1,ext2 external
```
