How bigdatateam5/marketscope-ai-powered-industry-segment-intelligence-platform works
<explanation>
1. Identify Project Type & Purpose
- MarketScope AI is a full‑stack, microservices‑based analytics platform for healthcare product insights.
- It combines a Streamlit frontend, a suite of FastAPI “MCP” microservices, AI agents, a RAG pipeline (Pinecone vector store), a Snowflake data warehouse, and batch ETL (Apache Airflow).
2. Analyze the File Structure
Top‑level files & folders:
• Dockerfile, docker‑compose.yml – container definitions and local orchestration
• .github/workflows/gce‑deploy.yml – CI/CD to Google Cloud
• run.py, fix_endpoints.py – unified entrypoints
• pyproject.toml, requirements.txt – dependency specs
Subdirectories:
• frontend/ – Streamlit UI (app.py + pages/*.py, styles.css)
• mcp_servers/ – five domain‑specific FastAPI services on ports 8000–8004
• agents/ – core AI orchestration (unified_agent.py, custom_mcp_client.py, utils)
• config/ – global settings, LLM client (litellm_service), paths
• Airflow/ – Dockerfile, DAGs for ingesting PDFs/JSON, chunking, loading vectors
• tests/ – unit tests for Pinecone, agents, etc.
• setup_snowflake.py, view_snowflake_tables.py – Snowflake integration scripts
3. Examine README for Additional Insights
- Features: market segmentation, query optimization (RAG over Kotler’s book), product comparison, sales & marketing analysis
- Dual run methods: run_servers.py (starter), or manually start FastAPI + Streamlit
- Deployed MCP endpoints on 8000–8004; Streamlit on 8501
- ETL for RAG knowledge, Snowflake for structured data
4. How to Create the System Design Diagram
a. Main Components (boxes)
- Client Browser
- Streamlit Frontend (frontend/app.py + pages/)
- Unified Orchestrator / API Gateway (run.py + agents/unified_agent.py)
- MCP Microservices (mcp_servers/*.py; one per domain)
- AI Agents & LLM Client (agents/, config/litellm_service.py)
- Vector Store (Pinecone)
- Data Warehouse (Snowflake)
- Object Storage (AWS S3)
- ETL Scheduler (Apache Airflow with DAGs in Airflow/dags/)
b. Relationships & Data Flows (arrows)
1. User → Streamlit UI → Unified Orchestrator
2. Orchestrator → domain‑specific MCP server (HTTP)
3. MCP server → Agent logic & LLM client → Pinecone / Snowflake / S3
4. Airflow DAGs fetch PDFs/JSON (Airflow/config) → chunk/parsing utils → load vectors → Pinecone
5. Snowflake ETL scripts ←→ Snowflake data warehouse
c. Architectural Patterns & Principles
- Microservices for domain separation (MCP servers)
- Orchestration layer decouples UI from services (unified_agent)
- RAG for knowledge retrieval (vector DB + chunked docs)
- Batch ETL with Airflow for ingest pipelines
- Containerization and Infrastructure as Code (Docker, docker‑compose, GitHub Actions → GCE)
d. Key Technologies & Frameworks
- Python 3.8+ / FastAPI / Streamlit
- Apache Airflow (DAGs + Dockerfile)
- Pinecone Vector Database
- Snowflake Data Warehouse
- AWS S3 for PDF storage
- Docker & Docker‑Compose
- GitHub Actions for CI/CD
5. Diagram‑Tailoring Guidelines for This Project
- Layer 1 (UI): Streamlit components (pages)
- Layer 2 (Orchestration): run.py → unified_agent → custom_mcp_client
- Layer 3 (Microservices): each MCP server as separate service box
- Layer 4 (Data Stores): Pinecone (vectors), Snowflake (tables), S3 (raw files)
- Layer 5 (ETL): Airflow scheduler + DAGs depicted as a batch pipeline
6. Diagram Notation Recommendations
- Clear labels: include folder/file references (e.g., “Market Segmentation MCP (mcp_servers/segment_mcp_server.py)”)
- Arrows: annotate with protocol (HTTP, gRPC, REST calls) and direction
- Color coding / Shapes:
• UI (green rectangle)
• Orchestrator (blue)
• Microservices (yellow)
• Data Stores (orange cylinder)
• ETL / Batch (purple parallelogram)
- Legend: define colors/shapes
Emphasis: Capture each folder/role as a distinct component. Show real file paths for traceability. Don’t under‑represent the Airflow layer or the multiple MCP servers—each is a standalone microservice.
</explanation>