How ricardoandresd6/move-files-process works
Project type and purpose
• This is a backend ETL (Extract-Transform-Load) Python application.
• Purpose: read records from an Excel file, filter duplicates, monitor system resources and internet connectivity, write results back to another Excel file, and insert records into a MySQL database.
File structure
• Top-level files:
• app.py → main Python ETL script
• main.ipynb → exploratory notebook (not part of production flow)
• requirements.txt → Python dependencies
• README.md → project documentation
• Folders:
• OriginFolder/ → input Excel files (MaestroFamiliasMin.xlsx)
• DestinationFolder/ → output Excel file and log artifacts
• No frontend or service-oriented subdirectories—this is a single-process script.
README for additional insights
• Key operations: duplicate check (by CODBARRA), CPU/memory threshold guard, internet-connectivity check, MySQL logging, continuous file-based checkpointing.
• Configurable parameters in app.py: Excel filename, DB credentials, resource-usage thresholds, wait interval.
a. Main components
1. Input Store (OriginFolder + Excel file)
2. ETL Process (app.py) with sub-components:
• Connectivity Checker
• Resource Monitor
• Duplicate Filter
• MySQL Connector
• Excel Writer
3. Output Store (DestinationFolder + Excel file)
4. External Service (Internet)
5. MySQL Database Server
b. Relationships and interactions
• Excel Reader → Duplicate Filter → if new record →
• → MySQL Connector → MySQL Server
• → Excel Writer → DestinationFolder
• Before each row: Resource Monitor → blocks/suspends if thresholds exceeded
• Before processing: Connectivity Checker → blocks if no internet
c. Architectural patterns & design principles
• Pipeline pattern: sequence of stages (read → filter → write)
• Checkpointing: file-based state persistence to avoid reprocessing
• Guard clauses: resource and connectivity checks decouple concerns
d. Key technologies & libraries
• pandas / openpyxl (or similar) for Excel I/O
• psutil for CPU/memory monitoring
• requests or socket check for internet connectivity
• mysql-connector-python or PyMySQL for DB access