How vishwasaichalla/social_media_analytics_dashboard-full_stack_data_visualization_platform works
Project Type & Purpose
• This is a full-stack social media analytics dashboard.
• Backend: Flask-based REST API, CSV ingestion, data transformation, MongoDB storage, APScheduler for mock data.
• Frontend: Streamlit dashboard with Plotly Express visualizations.
• Deployed via Docker Compose, optionally on AWS EC2.
File Structure Analysis
• Top-level:
• Dockerfile & docker-compose.yml: container definitions.
• backend/: Flask app (app.py), ingestion (data_ingest.py), storage (data_store.py), transformation (transformation.py), sample CSV.
• frontend/: Streamlit script (visualization.py).
• Documentation: PowerBI.md, data-ingestion.md, datastore-setup.md, visualization.md.
• CI: .github/workflows/super-linter.yml.
• requirements.txt for Python deps.
• Patterns:
• Client-server separation (frontend vs backend).
• ETL pipeline: data ingestion → transformation → storage.
• Scheduler for periodic data generation.
• Container orchestration via Docker Compose.
Key Insights from README
• Architecture section clearly defines components and interactions.
• Specifies technologies: Flask, MongoDB, APScheduler, Streamlit, Plotly.
• Data flow: initial ingestion, real-time API calls, visualization updates.
• Data Sources:
• Social_Media_Engagement.csv (initial load)
• Mock data generator (APScheduler)
• Backend Services:
• Data Ingest Service (data_ingest.py)
• Transformation Layer (transformation.py)
• Data Store (MongoDB)
• Flask API (app.py)
• Frontend:
• Streamlit Dashboard (visualization.py)
• Orchestration & Deployment:
• Docker Compose
• CI Pipeline (GitHub Actions)
• External / Infrastructure:
• AWS EC2 (optional)
• Elastic IP, Security Groups
1. CSV File → Data Ingest Service → Transformation Layer → MongoDB.
2. Scheduler → Data Ingest Service (every 30s) → Transformation → MongoDB.
3. Streamlit Frontend → REST API Calls → Flask API → MongoDB → API Response → Frontend.
4. Docker Compose orchestrates containers for MongoDB, backend, frontend.
5. CI Pipeline → Linter on code push.
c. Architectural Patterns & Principles:
• Layered Architecture: presentation (frontend), application (Flask), data (MongoDB).
• ETL Pipeline: ingestion → transformation → storage.
• Client-Server Model: Streamlit client consumption of REST API.
• Modularization: separate Python modules for ingest, transform, storage.
d. Key Technologies & Libraries:
• Python 3.8+, Flask, PyMongo
• APScheduler for background tasks
• Streamlit, Plotly Express
• Docker, Docker Compose
• GitHub Actions (super-linter)
• (Optional) AWS EC2 for production