# ricardoandresd6/move-files-process architecture

> Architecture diagram of the GitHub repository ricardoandresd6/move-files-process, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ricardoandresd6/move-files-process
- Repository: https://github.com/ricardoandresd6/move-files-process

## Overview

##### Project type and purpose
- This is a backend ETL (Extract-Transform-Load) Python application.
- Purpose: read records from an Excel file, filter duplicates, monitor system resources and internet connectivity, write results back to another Excel file, and insert records into a MySQL database.

##### File structure
- Top-level files:
- app.py         → main Python ETL script
- main.ipynb     → exploratory notebook (not part of production flow)
- requirements.txt → Python dependencies
- README.md      → project documentation
- Folders:
- OriginFolder/      → input Excel files (MaestroFamiliasMin.xlsx)
- DestinationFolder/ → output Excel file and log artifacts
- No frontend or service-oriented subdirectories—this is a single-process script.

##### README for additional insights
- Key operations: duplicate check (by CODBARRA), CPU/memory threshold guard, internet-connectivity check, MySQL logging, continuous file-based checkpointing.
- Configurable parameters in app.py: Excel filename, DB credentials, resource-usage thresholds, wait interval.

a. Main components
1. Input Store (OriginFolder + Excel file)
2. ETL Process (app.py) with sub-components:
- Connectivity Checker
- Resource Monitor
- Duplicate Filter
- MySQL Connector
- Excel Writer
3. Output Store (DestinationFolder + Excel file)
4. External Service (Internet)
5. MySQL Database Server
b. Relationships and interactions
- Excel Reader → Duplicate Filter → if new record →
- → MySQL Connector → MySQL Server
- → Excel Writer → DestinationFolder
- Before each row: Resource Monitor → blocks/suspends if thresholds exceeded
- Before processing: Connectivity Checker → blocks if no internet
c. Architectural patterns & design principles
- Pipeline pattern: sequence of stages (read → filter → write)
- Checkpointing: file-based state persistence to avoid reprocessing
- Guard clauses: resource and connectivity checks decouple concerns
d. Key technologies & libraries
- pandas / openpyxl (or similar) for Excel I/O
- psutil for CPU/memory monitoring
- requests or socket check for internet connectivity
- mysql-connector-python or PyMySQL for DB access

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration and Documentation
    Config["Config & Dependencies\n(requirements.txt)"]:::config
    Doc["Project Documentation\n(README.md)"]:::doc

    %% External Services
    Internet["Internet Service"]:::external
    MySQLDB[(MySQL Database Server)]:::db

    %% Storage
    InputStore["Input Store\nOriginFolder/MaestroFamiliasMin.xlsx"]:::storage
    OutputStore["Output Store\nDestinationFolder/MaestroFamilias.xlsx"]:::storage
    Logging["Checkpoint & Logs\nDestinationFolder/destination.txt"]:::storage

    %% ETL Process
    subgraph "ETL Process (app.py)"
        direction TB
        ConnectivityChecker["Connectivity Checker"]:::process
        ResourceMonitor["Resource Monitor"]:::process
        DuplicateFilter["Duplicate Filter"]:::process
        MySQLConnector["MySQL Connector"]:::process
        ExcelWriter["Excel Writer"]:::process
    end

    %% Data Flow
    Config -->|"loads parameters"| ConnectivityChecker
    Config --> ResourceMonitor

    InputStore -->|"reads Excel rows"| ConnectivityChecker
    ConnectivityChecker -->|success| ResourceMonitor
    ConnectivityChecker -.->|fail block| ResourceMonitor
    ResourceMonitor -->|within thresholds| DuplicateFilter
    ResourceMonitor -.->|threshold exceeded| ResourceMonitor

    DuplicateFilter -->|new record| MySQLConnector
    DuplicateFilter -->|new record| ExcelWriter
    DuplicateFilter -.->|duplicate| Logging

    MySQLConnector -->|SQL insert| MySQLDB
    ExcelWriter -->|writes Excel rows| OutputStore
    MySQLConnector --> Logging
    ExcelWriter --> Logging

    %% External connectivity check
    ConnectivityChecker -->|ping| Internet

    %% Click Events
    click Config "https://github.com/ricardoandresd6/move-files-process/blob/main/requirements.txt"
    click Doc "https://github.com/ricardoandresd6/move-files-process/blob/main/README.md"
    click InputStore "https://github.com/ricardoandresd6/move-files-process/blob/main/OriginFolder/MaestroFamiliasMin.xlsx"
    click ConnectivityChecker "https://github.com/ricardoandresd6/move-files-process/blob/main/app.py"
    click ResourceMonitor "https://github.com/ricardoandresd6/move-files-process/blob/main/app.py"
    click DuplicateFilter "https://github.com/ricardoandresd6/move-files-process/blob/main/app.py"
    click MySQLConnector "https://github.com/ricardoandresd6/move-files-process/blob/main/app.py"
    click ExcelWriter "https://github.com/ricardoandresd6/move-files-process/blob/main/app.py"
    click OutputStore "https://github.com/ricardoandresd6/move-files-process/blob/main/DestinationFolder/MaestroFamilias.xlsx"
    click Logging "https://github.com/ricardoandresd6/move-files-process/blob/main/DestinationFolder/destination.txt"

    %% Styles
    classDef storage fill:#D0E6FF,stroke:#0366D6,color:#0366D6
    classDef process fill:#DFFFE0,stroke:#228B22,color:#228B22
    classDef db fill:#FFF1D0,stroke:#DAA520,color:#DAA520
    classDef external fill:#F0F0F0,stroke:#AAA,color:#888
    classDef config fill:#FFE0E0,stroke:#D22,color:#D22
    classDef doc fill:#F5E6FF,stroke:#8A2BE2,color:#8A2BE2
```
