How kalpadas599/real-time-analysis-bi works
Project type and purpose
• This is a data-pipeline/full-stack microservice project that ingests data from external sources (Google Ads, Twitter), processes it with Spark and a spaCy-based sentiment analyzer, stores results in multiple backends (MongoDB, Snowflake, Firebase), and exposes a Flask API. It’s containerized (Dockerfile, docker-compose) with CI in GitHub workflows.
File structure
• Top-level:
• app.py / src/api/flask_api.py – Flask API entrypoint
• run_ingestion.py – orchestration script for data ingestion
• Dockerfile + docker-compose.yml – container definitions and service composition
• requirements.txt – Python dependencies
• mock_google_ads_data.json, power_bi_setup.md – auxiliary files
• en_core_web_sm… – bundled spaCy English model
• src/:
• api/ – Flask API code
• data_ingestion/ – modules to pull from Google Ads, Twitter, and push to Confluent Kafka
• data_processing/ – spark_processor for batch/stream processing
• ml/ – sentiment_analyzer running spaCy model
• storage/ – clients for Firebase, MongoDB, Snowflake
README
• (Empty in this case; all insights come from file tree.)
a. Main components:
1. External Data Sources (Google Ads API, Twitter API)
2. Data Ingestion Service (confluent_producer)
3. Message Broker (Kafka cluster)
4. Processing Layer (Spark Processor)
5. ML Layer (spaCy Sentiment Analyzer)
6. Storage Layer (MongoDB, Snowflake, Firebase)
7. API Layer (Flask API)
8. Orchestration/Containerization (Docker Compose)
9. CI/CD (GitHub Actions)
b. Relationships & interactions:
• Data Sources → Data Ingestion Service → Kafka
• Kafka → Spark Processor → ML Sentiment Analyzer → Storage Clients
• Storage Clients → Databases (MongoDB, Snowflake, Firebase)
• Flask API ↔ Databases (reads processed data)
• Docker-Compose runs ingest/process/api services in containers
• GitHub Actions triggers builds/tests on push
c. Architectural patterns & principles:
• Separation of concerns (ingest, process, ML, API)
• Event-driven architecture via Kafka
• Containerized microservices
• Polyglot storage (NoSQL, data warehouse, real-time DB)
d. Key technologies:
• Python, Flask, Spark, spaCy (en_core_web_sm)
• Confluent Kafka
• MongoDB, Snowflake, Firebase
• Docker, Docker-Compose
• GitHub Actions
Elements
• Blue for ingestion components
• Orange for processing/ML
• Green for storage
• Purple for API
• Gray for orchestration/CI/CD
• Rectangles for microservices