How aliyanishfaq/ai-market-researcher works
Project Type and Purpose
• This is a backend‑only, AI‑powered survey simulation service with a FastAPI REST interface.
• Purpose: generate persona‑based survey responses, run parallel simulations, perform deep analytics, and expose endpoints for “ask”, “survey/run”, and “research.”
File Structure
• server.py: the FastAPI entry point.
• ask_endpoint/: handles single‑persona Q&A via LLM.
• deep_research/: modules orchestrating multi‑step research pipelines.
• survey_meta_analysis/: modules for meta‑analysis prompts.
• Core scripts at top level (prompts.py, personas.py, schema.py, SurveyTypes.py, question_classifier.py, survey_simulation.py) implement:
• Persona Management
• Survey Simulation Engine
• LLM Inference orchestration
• Analytics Engine
• Data files (glassdoor.json, product‑reviews.json, intel_product_reviews.json) serve as input corpora.
• Requirements, Dockerfile, fly.toml, and .env configuration indicate containerized deployment.
README Insights
• Architecture overview lists five subsystems:
1. Persona Management
2. Survey Simulation
3. LLM Inference
4. Analytics Engine
5. API Server
• External dependencies: OpenAI, Azure OpenAI, Anthropic, AWS for potential storage/logging.
• No persistent database; data is read from JSON or passed through memory.
Main Components & Relationships
a. Components:
• Client (Market Researcher or UI)
• API Server (FastAPI in server.py)
• Persona Management Module (personas.py, schema.py, SurveyTypes.py)
• Survey Simulation Engine (survey_simulation.py, question_classifier.py)
• Ask Endpoint Module (ask_endpoint/)
• Deep Research Engine (deep_research/)
• Meta‑Analysis Module (survey_meta_analysis/)
• Analytics Engine (response_analytics.py, qualitative_analytics.py)
• Data Sources (static JSON files)
• LLM Providers (OpenAI/Azure/Anthropic – llminference.py, prompts.py)
• Configuration and Deployment (Docker, fly.toml, .env)
b. Interactions:
1. Client → API Server: HTTP POST /ask, /survey/run, /research
2. API Server routes request to appropriate module:
– /ask → ask_endpoint
– /survey/run → Survey Simulation Engine
– /research → Deep Research Engine
3. Within Simulation or Ask modules:
– Persona Management loads persona profile from JSON or in‑memory schema.
– Orchestrator distributes questions to multiple persona instances (parallel/concurrent).
– For each question, call LLM Inference → external LLM services → receive response.
4. Collected responses → Analytics Engine (response_analytics, qualitative_analytics, network & sentiment analysis).
5. Meta‑Analysis Module may consume analytics results for higher‑level insights.
6. Final JSON payload sent back to Client.
c. Architectural Patterns:
• Modular monolith: distinct folders for each domain.
• Pipeline pattern in deep_research (waterfall of prompt → LLM call → analysis → repeat).
• Parallel/concurrent processing of persona simulation.
• Adapter pattern for LLM providers (llminference.py abstracts OpenAI, Azure, Anthropic).
d. Key Technologies:
• Python 3.8+
• FastAPI for REST endpoints
• Asyncio or concurrent.futures for parallel simulations
• Azure/OpenAI/Anthropic APIs
• Docker + Fly.io for containerized deployment
• JSON files as data stores