How gauravssinha/petro_data_forecast works
Main Components
• Data Collection & Ingestion: Represent the AWS IoT Core (for sensor/device data input), along with the data ingestion scripts located in src/data/ingest_data.py.
• Data Storage: Include Amazon S3 as the central data repository where ingested and processed data are stored.
• ETL Operations: Highlight AWS Glue, which handles ETL tasks to process and transform the raw data.
• Model Development & Training: Use Amazon SageMaker to represent the environment for training machine learning models based on the scripts in src/models/train_model.py and forecasting logic in src/models/forecast.py.
• Visualization & Reporting: Illustrate Amazon QuickSight for dashboard visualization along with the script in src/visualization/create_dashboard.py.
• Monitoring & Logging: Represent Amazon CloudWatch for tracking performance, logs, and alerts.
Relationships and Data Flow
• Begin from the data ingestion layer (AWS IoT Core) feeding into Amazon Kinesis for real-time processing.
• Kinesis passes the processed data to both:
• Amazon S3 for storage and later use.
• AWS Glue for ETL tasks to further cleanse and prepare the data.
• The prepared data then flows into Amazon SageMaker where models are trained using the pipelines indicated in src/models.
• The trained model is used by the forecasting script (src/models/forecast.py) to generate predictions.
• Forecast results, along with processed metrics, are visualized in Amazon QuickSight.
• The entire infrastructure setup is initiated via the setup_infrastructure.py script, ensuring configuration and deployment of the needed AWS resources.
• CloudWatch monitors all these components to ensure timely alerts and logs are captured for performance and maintenance.
Architectural Patterns and Design Principles
• A modular, service-oriented architecture that clearly separates data ingestion, processing, ML model training, and visualization.
• Separation of concerns (data, models, visualization, utilities) which aligns with best practices seen in the folder structure.
• That AWS services act as managed microservices, which simplifies scaling and integration while maintaining clear boundaries.
Relevant Technologies and Frameworks
• AWS IoT Core, S3, Kinesis, Glue, SageMaker, QuickSight, and CloudWatch as core AWS services.
• Mention Python as the core language with libraries listed in requirements.txt, and external dependencies that may be used for data processing, model training, and visualization.