How handaru/data works
Project’s Main Domain Components
• Educational Content Modules – These include:
• Data Analytics (notebooks for exploratory data analysis, statistical methods, time series, etc.)
• Data Science (notebooks for machine learning, regression, classification, and model building – plus models saved under data_science/models)
• Data Visualization (Jupyter notebooks and integrations with tools like PowerBI and Tableau)
• Data Engineering / Colab – Notebooks on SQL and data pipelines under the colab/data_engineering folder.
• Practical Project Examples – Under the “buddy” folder:
• CRM Module (buddy/crm with a Python file and related PDFs)
• Risk Module (buddy/risk with Python environment and dependencies)
• SOP Module (buddy/sop, which includes further subcomponents like PDF image processing and Streamlit apps for interacting with content)
• Data Sets – Located in the "datasets" folder, these are core resources that feed the notebooks and examples.
• Configuration/Dependency Management – Files like requirements.txt, environment.yml, and .gitignore indicate versions and dependencies used across modules.
Out Relationships and Interactions
• Data Flow:
• Datasets (CSV files and test/train files) form the base input that feeds into the notebooks across data analytics and data science.
• Notebooks in data_science process data using models and produce artifacts, which may be visualized in the data_visualization or analytical reports.
• Modular Interaction:
• The buddy directory acts almost like a set of mini-projects or applications (CRM, risk, SOP) that could be seen as real-world applications of the data science and analytics learnings.
• The SOP module goes deeper by offering both static PDF manipulation (as seen with readmes and PDFs) and interactive dashboards through Streamlit apps.
• External Tools/Frameworks:
• Streamlit (for interactive web apps under buddy/sop/streamlit_apps) suggests a UI/UX presentation layer connected with backend processing.
• The notebooks incorporate various Python data science libraries, and the presence of environment and requirements files indicates dependency management and reproducibility.
Architectural Patterns & Design Principles
• Modular Structure: The directory separation into distinct domains encourages a modular view; treat each folder as an independent module with its own learning focus.
• Data-Driven Design: Emphasize that datasets are central, and everything from analysis to visualization and model building revolves around this core component.
• Reusability and Extensibility: Note that the separation of resources (e.g., standalone notebooks for different topics, self-contained mini-apps) promotes learning through hands-on examples, which can be extended or modified.
Technical Details to Annotate
• For components like buddy/sop, include annotations about integration with UI frameworks (like Streamlit) and PDF processing utilities.
• If relevant, note key libraries or frameworks as sub-labels (e.g., "scikit-learn, xgboost" for model building, "Pandas" for data manipulation, "Streamlit" for interactive apps).