How suhaspalani/big_data_instagram works
Project Type and Purpose
• This is a full-stack application specifically built to provide Instagram analytics.
• The README clearly reveals that key capabilities include real-time updates via WebSockets and performance monitoring of database operations.
File Structure
• The project file tree reveals three main high-level directories: “backend”, “consumer”, and “frontend”.
• The “backend” directory includes files such as main.py (likely launching the FastAPI service), kafka_producer.py (handling message queuing with Kafka), and configuration files.
• The “consumer” directory covers data processing roles with files like app.py, data_processor.py, and instagram_scraper.py (which integrates with Apify to scrape Instagram profiles). This service listens to Kafka and handles database interactions.
• The “frontend” directory houses a React.js app with components that display analytics, influencer forms, Kafka status, and performance metrics.
• Additionally, Docker-related files (Dockerfile in each component, docker-compose.yml) indicate that containerization is employed to standardize deployments.
Extract Architectural Insights from the README
• The README highlights key components like FastAPI, Kafka, MongoDB, DynamoDB, Apify integration, and a React frontend.
• It details the data flow: from scraping Instagram profiles (consumer fetching data via API calls) to processing messages via Kafka, and finally serving data to the frontend using WebSockets.
• The README also underlines the importance of scalability and performance monitoring.
a. Main Components:
• Frontend: React.js application for data visualization.
• Backend: FastAPI service handling API requests and interfacing with Kafka.
• Consumer: Python service that listens to Kafka topics, scrapes Instagram data, and processes/stores it.
• Databases: Two storage systems – MongoDB (local) as well as AWS DynamoDB (cloud).
• External Services: Kafka (and ZooKeeper), Apify for scraping, and possibly AWS services for DynamoDB.
• Docker & Docker Compose: Containerization and orchestration layer ensuring services run in a coordinated fashion.
b. Relationships and Data Flow:
• The Frontend (React app) communicates with the Backend (FastAPI) via API calls and connects via WebSockets for real-time data updates.
• The Backend sends messages to Kafka. These messages include Instagram profile analysis requests.
• The Consumer service listens to Kafka topics. On receiving messages, it uses Apify to scrape Instagram profiles, processes the data, and writes to either MongoDB or DynamoDB.
• API calls from Frontend to Backend.
• Message flow from Backend (Kafka Producer) to Kafka.
• Kafka messages being consumed by the Consumer service.
• Real-time updates flowing back from Backend (via WebSockets) to Frontend.
c. Architectural Patterns and Technologies:
• Microservices: Backend and Consumer are separate services communicating asynchronously via Kafka.
• Containerization: Docker and docker-compose ensure modular deployment.
• Data-driven pipeline: Messaging queue (Kafka) decouples scraping/processing from API layer.
• Scalability: Use of Kafka and cloud-based DynamoDB shows the design’s readiness to scale.
• Use of modern frameworks: FastAPI, React.js, and integration with Apify.
d. Relevant Technologies:
• Backend: FastAPI (Python), Kafka (with a producer), and configuration management via config.py.
• Consumer: Python with modules for data processing, Instagram scraping using Apify, and database handling.
• Frontend: React.js with components handling different analytics UI functions.
• Databases: MongoDB and DynamoDB.
• Orchestration: Docker, docker-compose, and external services (Kafka, ZooKeeper).