How jonasluis/intuitive-care works
• Data Collection Module:
• Web Scraping (backend/scraper.py): Automates the collection of PDF links from the ANS website.
• Downloaders (backend/downloader.py and backend/ftp_downloader.py): Responsible for downloading PDFs and data files via FTP.
• Data Processing Module:
• Data Transformation (backend/data_transformer.py): Handles processing, cleaning, and converting data (e.g., PDF to CSV) after collection.
• Compression (backend/compressor.py): Compresses files after processing.
• CSV Checks (backend/check_csv.py): Validates or checks CSV integrity.
• Database Operations Module:
• Scripts (backend/database/): SQL files (create_tables.sql and import_and_analyze.sql) for initializing the MySQL database schema and managing data analysis.
• API Module:
• API Endpoints (backend/api.py): Implements RESTful endpoints using FastAPI.
• API Runner (backend/run_api.py): Script to start the FastAPI server.
• Main Orchestrator (backend/main.py): Coordinates the data collection and transformation processes through command-line arguments that trigger specific routines.
• Frontend Module:
• User Interface (frontend/index.html and frontend/src/App.vue): Main view layer in Vue.js to present search and data views.
• Entry Point (frontend/src/main.js): Initializes the Vue application.
• Service Layer (frontend/src/services/operadoraService.js): Handles client-side API requests using Axios.
• External Systems:
• MySQL Database: Holds the transformed data for operations and analytics.
• External Data Sources: ANS website and FTP endpoints for obtaining data.
• Postman Collection: Documentation for API endpoints.
Interactions and Relationships
• Data Flow:
• The API layer (FastAPI) then interacts with the database to serve data requests.
• The frontend Vue.js application communicates with the API endpoints (via Axios calls) to fetch and display data.
• Dependencies:
• The orchestration (backend/main.py) relies on separate modules (scraper, downloader, transformer) that eventually supply data into the database.
• The API (backend/api.py) acts as a middleware, providing RESTful endpoints to the Vue.js frontend.
• Configuration settings in config.py ensure that environmental parameters (like external URLs and API endpoints) are correctly injected into the system.
Architectural Patterns and Design Principles
• Layered Architecture: Clearly separate data collection, processing, and presentation layers.
• Modular Decomposition: Each Python script and package in the backend encapsulates specific functionalities (scraping, downloading, transformation, etc.).
• API-Driven Design: Communication between the frontend (Vue.js) and backend (FastAPI) is through well-defined RESTful APIs.
• Separation of Concerns: Different components (data manipulation vs. user interface) are isolated to simplify maintenance and updates.