# mahpatil/python-notebooks architecture

> Architecture diagram of the GitHub repository mahpatil/python-notebooks, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mahpatil/python-notebooks
- Repository: https://github.com/mahpatil/python-notebooks

## Overview

This project is essentially a curated collection of Jupyter notebooks centered on various data, AI, and ML tasks. The overall purpose is personal documentation and experimentation across multiple domains (e.g., data ingestion, processing, model analysis, visualization) with notebooks organized by subject matter (such as Atlassian integrations, billing data processing, Cassandra data pipelines, Covid19 analysis via BigQuery, etc.).

##### Main Components
- Data Sources and Sinks: Highlight external services or data sources mentioned in the notebooks (e.g., Atlassian APIs, AWS Billing, Cassandra databases, BigQuery datasets, Google Firestore, etc.).
- Processing/Analysis Engines: Note the recurring use of Jupyter as a development and experimentation environment, along with any data processing libraries or frameworks that are implicitly used in the notebooks.

##### Relationships and Interactions
- Dependencies: Even if these notebooks work independently, group together those that share a theme—for example, Atlassian notebooks might be connected, indicating steps from data extraction (Bitbucket, Jira) to transformation (epic lists, transition states).
- Integration Points: Mark any notebooks that show interconnections, such as those converting Excel files or comparing JSON outputs, as integration or bridging components.

##### Architectural Patterns and Design Principles
- Modularity: Emphasize that each directory/subsystem is a self-contained module meant to handle a specific data or analysis task.
- Separation of Concerns: The organization of notebooks by domain (e.g., billing vs. covid19 analyses) shows a clear separation between different functionalities.
- Reusability and Experimentation: The notebooks act as individual experiments or processes, which means the architecture is flexible and meant for iterative analysis rather than a fixed production pipeline.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Processing/Analysis Engine"
        J("Jupyter Environment (init.ipynb)") 
    end

    subgraph "Data Acquisition & Integration"
        A1("Atlassian – Data extraction & API integration")
        A2("Billing – AWS Billing & Freeagent extraction")
        A3("Google Firestore – Data import")
    end

    subgraph "Data Storage / ETL"
        B1("Cassandra – Data pipelines & transformation")
        B2("Covid19 BigQuery – ETL for Covid data")
    end

    subgraph "Analysis & Modeling"
        C1("Linear Regression – Statistical analysis")
        C2("Math – Standard deviation & calculations")
        C3("PDF-GPT – NLP and PDF processing")
        C4("Resume – Resume processing via NLP")
        C5("Text Comparison – Regex and text analytics")
        C6("Social Network History – Social media analysis")
    end

    subgraph "Supporting Utilities"
        D1("Excel – File conversion & data output")
        D2("JSON Compare – JSON file comparisons")
        D3("Diagrams – Visualization & diagram generation")
    end

    %% Connections from Engine to Acquisition
    J -->|"triggers"| A1
    J -->|"triggers"| A2
    J -->|"triggers"| A3

    %% Connections from Acquisition to Data Storage/ETL
    A1 -->|"sends data"| B1
    A2 -->|"sends data"| B1
    A3 -->|"sends data"| B2

    %% Connections from Data Storage/ETL to Analysis & Modeling
    B1 -->|"prepares"| C1
    B1 -->|"prepares"| C2
    B1 -->|"prepares"| C5
    B1 -->|"prepares"| C6
    B2 -->|"prepares"| C3
    B2 -->|"prepares"| C4

    %% Connections from Analysis & Modeling to Supporting Utilities
    C4 -->|"outputs"| D1
    C5 -->|"outputs"| D2
    C6 -->|"outputs"| D3

    %% Click Events
    click J "https://github.com/mahpatil/python-notebooks/blob/master/init.ipynb"
    click A1 "https://github.com/mahpatil/python-notebooks/tree/master/atlassian/"
    click A2 "https://github.com/mahpatil/python-notebooks/tree/master/billing/"
    click A3 "https://github.com/mahpatil/python-notebooks/tree/master/google-firestore/"
    click B1 "https://github.com/mahpatil/python-notebooks/tree/master/cassandra/"
    click B2 "https://github.com/mahpatil/python-notebooks/tree/master/covid19-bigquery/"
    click C1 "https://github.com/mahpatil/python-notebooks/tree/master/linear-regression/"
    click C2 "https://github.com/mahpatil/python-notebooks/tree/master/math/"
    click C3 "https://github.com/mahpatil/python-notebooks/tree/master/pdf-gpt/"
    click C4 "https://github.com/mahpatil/python-notebooks/tree/master/resume/"
    click C5 "https://github.com/mahpatil/python-notebooks/tree/master/text-comparison/"
    click C6 "https://github.com/mahpatil/python-notebooks/tree/master/social-network-history/"
    click D1 "https://github.com/mahpatil/python-notebooks/tree/master/excel/"
    click D2 "https://github.com/mahpatil/python-notebooks/tree/master/json-compare/"
    click D3 "https://github.com/mahpatil/python-notebooks/tree/master/diagrams/"

    %% Styles
    class J engine
    class A1,A2,A3 acquisition
    class B1,B2 storage
    class C1,C2,C3,C4,C5,C6 analysis
    class D1,D2,D3 utilities

    classDef engine fill:#fc9,stroke:#333,stroke-width:2px;
    classDef acquisition fill:#f9c,stroke:#333,stroke-width:2px;
    classDef storage fill:#9f9,stroke:#333,stroke-width:2px;
    classDef analysis fill:#ff9,stroke:#333,stroke-width:2px;
    classDef utilities fill:#9ff,stroke:#333,stroke-width:2px;
```
