How shayansm2/tech-career-explorer works
Main Components
• Data Ingestion (Crawlers & Data Loaders):
– Crawler modules (found under src/crawlers and google-scripts) are responsible for scanning listing and detail pages.
– Data loaders (see src/data_loaders) kick off crawling, invoking crawlers to retrieve raw job data.
• Data Transformation:
– DBT components (located under src/dbt/job_positions) execute transformation on the raw data to create final tables (fact_positions and fact_metadata). Also include pipelines (src/pipelines) that coordinate DBT runs and other transformation processes.
• Data Exporting:
– Data exporter modules (under src/data_exporters) take the cleaned and transformed data to load them into the Data Warehouse.
• Data Warehouse:
– PostgreSQL serves as the central repository for production (with the backup in DuckDB). Indicate the two main tables (fact_metadata and fact_positions) along with techniques like partitioning and clustering.
• Workflow Orchestration:
– Mage-AI is used to manage ETL jobs; include its integration and how it coordinates batch processing.
• Dashboard:
– Metabase connects to the warehouse and provides visualization/analytics.
• Additional Components:
– Configuration files and YAML files (src/configs, io_config.yaml, metadata files) guide execution and transformation rules.
– Custom modules, extensions, and schema definitions add to custom logic and data definitions.
Relationships and Interactions
– From external web sources (LinkedIn, Glassdoor, Relocate) to the Crawlers.
– From Crawlers to Data Loaders and then to Data Exporters performing minor cleaning and formatting tasks.
– From Data Exporters to the Data Warehouse (PostgreSQL) with a “Backup/Dev” branch to DuckDB.
– The orchestration platform (Mage-AI) should be centralized showing how it triggers and monitors ingestion and transformation pipelines.
Architectural Patterns and Design Principles
• The architecture combines an ELT approach with selective ETL processes:
– Crawlers extract and initially transform data before loading.
– DBT handles heavy transformation, following batch processing best practices.
• Separation of concern is evident in distinct directories for crawlers, loaders, exporters, pipelines, and transformations.
• Modularity: Each component (crawlers, pipelines, dbt projects) is self-contained for scalability and maintainability.
• Use of configuration files and a clear modular structure supports extensibility.
Detail and Clarity
• Be very detailed in subdividing the modules especially given the rich file structure. For instance, highlight that the DBT process includes separate folders for staging and core models.