How mahpatil/python-notebooks works
This project is essentially a curated collection of Jupyter notebooks centered on various data, AI, and ML tasks. The overall purpose is personal documentation and experimentation across multiple domains (e.g., data ingestion, processing, model analysis, visualization) with notebooks organized by subject matter (such as Atlassian integrations, billing data processing, Cassandra data pipelines, Covid19 analysis via BigQuery, etc.).
Main Components
• Data Sources and Sinks: Highlight external services or data sources mentioned in the notebooks (e.g., Atlassian APIs, AWS Billing, Cassandra databases, BigQuery datasets, Google Firestore, etc.).
• Processing/Analysis Engines: Note the recurring use of Jupyter as a development and experimentation environment, along with any data processing libraries or frameworks that are implicitly used in the notebooks.
Relationships and Interactions
• Dependencies: Even if these notebooks work independently, group together those that share a theme—for example, Atlassian notebooks might be connected, indicating steps from data extraction (Bitbucket, Jira) to transformation (epic lists, transition states).
• Integration Points: Mark any notebooks that show interconnections, such as those converting Excel files or comparing JSON outputs, as integration or bridging components.
Architectural Patterns and Design Principles
• Modularity: Emphasize that each directory/subsystem is a self-contained module meant to handle a specific data or analysis task.
• Separation of Concerns: The organization of notebooks by domain (e.g., billing vs. covid19 analyses) shows a clear separation between different functionalities.
• Reusability and Experimentation: The notebooks act as individual experiments or processes, which means the architecture is flexible and meant for iterative analysis rather than a fixed production pipeline.