How rosdatatech/rdt works
Project type and purpose
• This is a research‑oriented data‑analysis and reporting tool rather than a classic web full‑stack app.
• It ingests curricula/program data from university websites (via Parsers), pulls job postings (rdt_hhru), stores everything in PostgreSQL, runs ML/NLP analysis (rdt_ml), and produces interactive views (PyQt5 in rdt_db_viewer) and PDF reports (rdt_pdf).
File structure
• Top‑level “Parsers/” folder contains domain‑specific ETL scripts. Each subfolder (Geology, IT, etc.) has …parser.py (scrapes/parses HTML), db_uploader.py (pushes parsed records into the DB).
• “main/” holds the core orchestration:
• main.py – entry point coordinating the pipeline
• rdt_database.py – DB connectivity and schema setup
• rdt_hhru.py – job‑listing scraper
• rdt_ml.py – analysis pipeline (using scikit‑learn, sentence‑transformers, gensim, numpy)
• rdt_pdf.py – report generation
• rdt_db_viewer.py – PyQt5 GUI for exploring stored data
• requirements.txt lists all dependencies; README describes the tech stack and high‑level flow.
Core components
a. Data Sources (external)
• University websites (various domains)
• Job portal (hh.ru or equivalent)
b. Ingestion & ETL Layer (“Parsers/” + rdt_hhru)
• Parsing scripts and download_func modules
• db_uploader scripts
c. Storage Layer
• rdt_database module for ORM/connection
d. Processing/Analytics Layer
• rdt_ml module — NLP embedding (sentence‑transformers), topic modeling (gensim), classification/regression (scikit‑learn)
e. Presentation & Reporting Layer
• rdt_db_viewer (PyQt5 GUI)
• rdt_pdf (PDF report generator)
f. Orchestrator
• main.py – sequences ingestion, analysis, and reporting
Relationships & Interactions
• Data flow from external sources → Parsers & rdt_hhru → PostgreSQL
• rdt_database sits between PostgreSQL and all internal modules needing data
• rdt_ml reads from DB, writes analysis results back to DB or to a report object
• rdt_db_viewer reads from DB to display interactive dashboards
• rdt_pdf reads from DB and rdt_ml outputs to generate final PDF
• main.py invokes each component in sequence (ingest → analyze → present)
Architectural patterns & principles
• Modular ETL: one submodule per domain, each with clear responsibility (parse vs. upload)
• Layered architecture: Ingestion, Storage, Processing, Presentation
• Single orchestrator entry point to simplify end‑to‑end runs
• Separation of concerns: database connectivity isolated in rdt_database
Final advice
• Be exhaustive in mapping every script/module to one of the layers.
• Don’t overcomplicate: it’s better to show many small components than to hide them in a monolith.