How atharvaj1425/distributed_csv_processor works
1. The Master Server (in the “master” directory) that manages CSV uploads, task coordination, and real‐time updates via Flask and SocketIO.
2. Worker Nodes (in the “worker” directory) that process the CSV file tasks retrieved from RabbitMQ.
3. A Dashboard (in the “dashboard” directory) built with Streamlit for visualization and user interaction.
Main Components
• Dashboard: “Streamlit Dashboard” that provides file upload and visualization.
• Master Server: Contains Flask “app.py”, CSV handling (“csv_processor.py”), RabbitMQ client (“rabbitmq_handler.py”), and SocketIO integration (“socket_handler.py”).
• Worker Nodes: Represented by “worker.py” (and its associated CSV processing logic) that operate in parallel.
• External Services: RabbitMQ for message queuing and optionally clients using SocketIO for real-time updates.
Relationships and Data Flows
• The Dashboard sends CSV file uploads to the Master Server.
• The Master Server enqueues tasks into RabbitMQ for distribution.
• Worker Nodes subscribe to RabbitMQ, pick up tasks, process the CSV data, and potentially send results back.
• The Master Server uses Flask-SocketIO to push real-time updates (e.g., process statuses) to the Dashboard.
Architectural Patterns and Design Principles
• Distributed processing: Separate master-worker paradigm where the master delegates tasks to scalable worker nodes.
• Real-time communications: SocketIO for push-based updates enhances interactivity.
• Message deduplication and idempotency: Note these design choices (as explained in the README) to underline fault tolerance.
• Scalability: Illustrate how new worker nodes can be added to the system without significant reconfiguration.
Incorporate Key Technologies and Frameworks
• Technologies to include:
• Flask (for the Master Server)
• Flask-SocketIO (for real-time updates)
• RabbitMQ (for task queue and message broker service)
• Streamlit (for the Dashboard UI)
• Python libraries such as pandas used in processing CSV files
• Optionally, indicate that the project uses Python’s eventlet or asynchronous patterns (as suggested by the README) for handling concurrent tasks.
Putting It All Together
• In the middle, detail the Master Server with its internal components to reflect its multifaceted role.
• Optionally, include a “Test/Integration” component (referencing run_tests.py and test_integration.py) to show how the system’s integration is verified if it adds context for deployment considerations.
This detailed representation will assist any engineer in understanding the data flows, component responsibilities, and where scalability and fault-tolerant designs are implemented in your system.