How youssef-harby/parquet-edge works
Project Type & Purpose
• This is a sensor data collection and processing application designed to run on a Linux device (preferably a Raspberry Pi). Its main task is to collect data from the Enviro+ HAT (covering sensors like temperature, pressure, humidity, etc.), save this information in an efficient Parquet format, and then synchronize these files to an S3 bucket.
• The project also includes a daily aggregation component and is set up to run via cron jobs and GitHub workflows.
Analyzing the File Structure
• Top-level Files and Directories:
• main.py: The primary script responsible for initiating sensor data collection.
• scripts/ directory: Contains additional helper scripts, such as daily_aggregation.py for data aggregation tasks.
• sync_to_s3.sh and start_data_collection.sh: Shell scripts that manage data synchronization and data collection startup, respectively.
• crontab_config.txt: Provides instructions for scheduling periodic jobs (starting data collection on boot, syncing to S3 every 5 minutes, etc.).
• .github/workflows: Includes a GitHub Actions YAML file (python-daily-aggregation.yml) for automated or CI/CD-based aggregation.
• pyproject.toml and .python-version: Define the Python project environment and dependencies.
• The file tree indicates that while the application logic is quite centralized (mostly in main.py and daily_aggregation.py), there is clear separation for data collection, aggregation, and synchronization components.
• Main Components to Represent:
• Data Collection Service (main.py): This module interfaces directly with the sensors.
• Data Storage: Illustrate the local file system or output directory where Parquet files are written (organized with minute-aligned timestamps).
• Data Aggregation Service (scripts/daily_aggregation.py & the GitHub workflow): A processing layer that periodically aggregates the collected data.
• Data Synchronization Component (sync_to_s3.sh): A script/action that syncs the locally stored Parquet files to an AWS S3 bucket.
• Cron Job Scheduler (crontab_config.txt): Represent the scheduling mechanism that triggers both data collection on boot and periodic syncing.
• Continuous Integration / Deployment (GitHub Actions workflow): Optionally, show how automation is managed through the .github/workflows directory.
• Auxiliary Setup Components: Configuration files (pyproject.toml, .python-version) that ensure the correct runtime and dependency management.
• Relationships and Interactions:
• The "Data Collection Service" writes output to local storage as Parquet files.
• The "Data Aggregation Service" reads these Parquet files for further processing or aggregation.
• The "Data Synchronization Component" (via sync_to_s3.sh) transfers/backs up the locally stored files to AWS S3.
• That the "Cron Job Scheduler" initiates and controls the timing for both main.py and sync_to_s3.sh.
• Architectural Patterns & Design Principles:
• Separation of concerns between data collection, processing, and synchronization.
• Scheduling (using cron jobs) as a key orchestration mechanism.
• That while the processing isn’t based on a heavy framework like MVC, the modular separation (different scripts for distinct functionality) is central to the design.
• Technologies include Python (with its ecosystem and dependencies), shell scripting, and AWS (for S3 storage).