# youssef-harby/parquet-edge architecture

> Architecture diagram of the GitHub repository youssef-harby/parquet-edge, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/youssef-harby/parquet-edge
- Repository: https://github.com/youssef-harby/parquet-edge

## Overview

##### Project Type & Purpose
- This is a sensor data collection and processing application designed to run on a Linux device (preferably a Raspberry Pi). Its main task is to collect data from the Enviro+ HAT (covering sensors like temperature, pressure, humidity, etc.), save this information in an efficient Parquet format, and then synchronize these files to an S3 bucket.
- The project also includes a daily aggregation component and is set up to run via cron jobs and GitHub workflows.

##### Analyzing the File Structure
- Top-level Files and Directories:
- main.py: The primary script responsible for initiating sensor data collection.
- scripts/ directory: Contains additional helper scripts, such as daily_aggregation.py for data aggregation tasks.
- sync_to_s3.sh and start_data_collection.sh: Shell scripts that manage data synchronization and data collection startup, respectively.
- crontab_config.txt: Provides instructions for scheduling periodic jobs (starting data collection on boot, syncing to S3 every 5 minutes, etc.).
- .github/workflows: Includes a GitHub Actions YAML file (python-daily-aggregation.yml) for automated or CI/CD-based aggregation.
- pyproject.toml and .python-version: Define the Python project environment and dependencies.
- The file tree indicates that while the application logic is quite centralized (mostly in main.py and daily_aggregation.py), there is clear separation for data collection, aggregation, and synchronization components.

- Main Components to Represent:
- Data Collection Service (main.py): This module interfaces directly with the sensors.
- Data Storage: Illustrate the local file system or output directory where Parquet files are written (organized with minute-aligned timestamps).
- Data Aggregation Service (scripts/daily_aggregation.py & the GitHub workflow): A processing layer that periodically aggregates the collected data.
- Data Synchronization Component (sync_to_s3.sh): A script/action that syncs the locally stored Parquet files to an AWS S3 bucket.
- Cron Job Scheduler (crontab_config.txt): Represent the scheduling mechanism that triggers both data collection on boot and periodic syncing.
- Continuous Integration / Deployment (GitHub Actions workflow): Optionally, show how automation is managed through the .github/workflows directory.
- Auxiliary Setup Components: Configuration files (pyproject.toml, .python-version) that ensure the correct runtime and dependency management.

- Relationships and Interactions:
- The "Data Collection Service" writes output to local storage as Parquet files.
- The "Data Aggregation Service" reads these Parquet files for further processing or aggregation.
- The "Data Synchronization Component" (via sync_to_s3.sh) transfers/backs up the locally stored files to AWS S3.
- That the "Cron Job Scheduler" initiates and controls the timing for both main.py and sync_to_s3.sh.

- Architectural Patterns & Design Principles:
- Separation of concerns between data collection, processing, and synchronization.
- Scheduling (using cron jobs) as a key orchestration mechanism.
- That while the processing isn’t based on a heavy framework like MVC, the modular separation (different scripts for distinct functionality) is central to the design.
- Technologies include Python (with its ecosystem and dependencies), shell scripting, and AWS (for S3 storage).

## Diagram (Mermaid)

```mermaid
graph TD
    %% External Hardware
    SENSOR["Sensor Hardware (Enviro+ HAT)"]:::sensor

    %% Cron Scheduler
    CRON["Cron Job Scheduler"]:::cron

    %% Data Storage and Cloud
    LS["Local Storage (Parquet Files)"]:::storage
    S3["AWS S3 Bucket"]:::cloud

    %% Data Synchronization Component
    DSYNC["Data Synchronization"]:::sync

    %% Data Collection Module
    subgraph "Data Collection Module"
        DCS_main["Data Collection Service"]:::collection
        DCS_start["Start Data Collection"]:::collection
    end

    %% Data Aggregation Module
    subgraph "Data Aggregation Module"
        DAS_script["Data Aggregation Service"]:::aggregation
        DAS_workflow["GitHub Workflow Aggregation"]:::aggregation
    end

    %% Auxiliary Setup
    AUX1["Auxiliary Setup"]:::aux
    AUX2["Python Version Setup"]:::aux

    %% Connections
    SENSOR -->|"feeds"| DCS_main
    CRON -->|"schedules"| DCS_main
    CRON -->|"schedules"| DCS_start
    DCS_main -->|"writesTo"| LS
    LS -->|"feeds"| DAS_script
    DAS_workflow -->|"triggers"| DAS_script
    LS -->|"feeds"| DSYNC
    CRON -->|"schedules"| DSYNC
    DSYNC -->|"syncsTo"| S3

    %% Click Events
    click DCS_main "https://github.com/youssef-harby/parquet-edge/blob/main/main.py"
    click DCS_start "https://github.com/youssef-harby/parquet-edge/blob/main/start_data_collection.sh"
    click DAS_script "https://github.com/youssef-harby/parquet-edge/blob/main/scripts/daily_aggregation.py"
    click DAS_workflow "https://github.com/youssef-harby/parquet-edge/blob/main/.github/workflows/python-daily-aggregation.yml"
    click DSYNC "https://github.com/youssef-harby/parquet-edge/blob/main/sync_to_s3.sh"
    click CRON "https://github.com/youssef-harby/parquet-edge/blob/main/crontab_config.txt"
    click AUX1 "https://github.com/youssef-harby/parquet-edge/blob/main/pyproject.toml"
    click AUX2 "https://github.com/youssef-harby/parquet-edge/blob/main/.python-version"

    %% Styles
    classDef sensor fill:#AED6F1,stroke:#1B4F72,stroke-width:2px;
    classDef collection fill:#A9DFBF,stroke:#196F3D,stroke-width:2px;
    classDef aggregation fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef sync fill:#D2B4DE,stroke:#6C3483,stroke-width:2px;
    classDef storage fill:#FADBD8,stroke:#922B21,stroke-width:2px;
    classDef cron fill:#F5CBA7,stroke:#D35400,stroke-width:2px;
    classDef aux fill:#D6EAF8,stroke:#2874A6,stroke-width:2px;
    classDef cloud fill:#FCF3CF,stroke:#B7950B,stroke-width:2px;
```
