How ggiannarakis/causal-jobs works
Main components
• Email Source and Gmail API Integration – where the raw job alert emails reside. Note that the extract.py script reads emails from the Gmail account’s “causal-jobs” folder.
• ETL Pipeline:
– Extraction: represented by extract.py which connects to the Gmail API.
– Transformation: captured by transform.py which converts raw email data into a structured Pandas dataframe.
– Loading: load.py which pushes the transformed data into the PostgreSQL database.
• Data Store – the PostgreSQL database running inside a Docker container (including the one table that holds all job alerts).
• Analysis Component – Analysis.ipynb (Jupyter Notebook) which runs the data analysis and generates a fresh report.
• Deployment and Reporting – index.html generated by the analysis, pushed via the causal-analysis-weekly-powershell.ps1 (or equivalent Bash script) to GitHub Pages.
• Scheduling and Alerting – daily and weekly scheduling is achieved through PowerShell (or Cron) scripts. In addition, there is a mention of a potential send_email.py script that can be used for alerts when the pipeline fails.
Relationships and interactions
• The ETL Pipeline should then be shown populating the Dockerized PostgreSQL database.
• Finally, illustrate the flow from the Analysis component to the deployment stage (GitHub pages hosting index.html) using the weekly PowerShell/Bash script.
Architectural patterns and technologies
• The ETL pattern is a key design decision.
• You’re using Python-based scripting (extract.py, transform.py, load.py) integrating with the Gmail API.
• Docker is central to hosting the PostgreSQL database.
• GitHub Pages is used for static website hosting of the analysis report.
• Scheduling is implemented externally via system-level scripting (PowerShell/Bash) and Task Scheduler or Cron.
• Including the flow from raw email extraction through transformation and loading as a “pipeline” block to emphasize the separation of concerns.
Detail and logical grouping
• Distinguish between daily operations (ETL scheduled scripts) and weekly operations (analysis and index.html deployment).
• Configuration files like requirements.txt that help set up the environment, and mention the role that Docker plays (with the PostgreSQL container) if necessary.