How ahmedennaifer/iot-streaming-platform works
Main components
• Sensor Generator (Python):
– Under src/main/generator, you have modules for “producer” (with producer.py and run_both.sh) and “sensors” (with specific sensor types defined in enumerations and sensor modules). This part simulates or collects sensor data.
• Data Consumer (Scala):
– Located under src/main/scala/consumer, this contains configuration (SparkConfig.scala and config.scala), processing (MetricsProcessor.scala), and a main Spark consumer (sparkConsumer.scala). There is also a database interaction layer (database.scala and DatabaseWriter.scala) that likely persists processed metrics.
• Configuration & Resources:
– The application.conf file in resources and various build configuration files (build.sbt, pyproject.toml) define environment setup and project dependencies.
• Deployment & Containerization:
– Dockerfiles (both in docker/python and docker/scala) and docker-compose.yml highlight a multi-container deployment strategy.
• Monitoring:
– The README includes images of the “Platform Architecture” and the “Grafana Dashboard”, suggesting that monitoring, alerting, and dashboard visualization are part of the deployed system.
Relationships and interactions
• Data Flow Starts in the Python Sensor Generator:
– The sensor modules (e.g., sensors/enums/temperature.py, sensors/humidity.py) simulate or read sensor data.
– The producer (producer.py) aggregates or dispatches this sensor data. The run_both.sh script suggests a coordinated start or simultaneous running of sensor data generation.
• Data Hand-Off to the Scala Consumer:
– The consumer (sparkConsumer.scala) receives this data – potentially via a queue, file drop, or another transport layer (even if not immediately obvious from the structure, the separation suggests a producer/consumer paradigm).
– Once ingested, the MetricsProcessor processes the data and passes the results on.
• Persistent Storage and Visualization:
– The DatabaseWriter and database.scala components demonstrate that processed data is written to a database.
• Deployment and Orchestration:
– Docker files indicate containerization of both the Python producer and Scala consumer components.
– Docker-compose orchestrates these containers, defining how they interact in production.
Architectural patterns and design principles
• Separation of Concerns:
– The generator and consumer are clearly separated into different modules and possibly different containers.
• Modular and Extensible Design:
– The use of Scala (with Spark) for data processing creates a scalable cloud-based processing pipeline.
– The Python generator is likely designed to be easily extensible (additional sensor types can be added without major rewrites).
• Containerization:
– Having dedicated Docker setups for each component ensures a reproduceable deployment environment.
• Monitoring and Observability:
– The integration with Grafana demonstrates the emphasis on real-time monitoring of key metrics.