# aliyanishfaq/ai-market-researcher architecture

> Architecture diagram of the GitHub repository aliyanishfaq/ai-market-researcher, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aliyanishfaq/ai-market-researcher
- Repository: https://github.com/aliyanishfaq/ai-market-researcher

## Overview

##### Project Type and Purpose
- This is a backend‑only, AI‑powered survey simulation service with a FastAPI REST interface.
- Purpose: generate persona‑based survey responses, run parallel simulations, perform deep analytics, and expose endpoints for “ask”, “survey/run”, and “research.”

##### File Structure
- server.py: the FastAPI entry point.
- ask_endpoint/: handles single‑persona Q&A via LLM.
- deep_research/: modules orchestrating multi‑step research pipelines.
- survey_meta_analysis/: modules for meta‑analysis prompts.
- Core scripts at top level (prompts.py, personas.py, schema.py, SurveyTypes.py, question_classifier.py, survey_simulation.py) implement:
- Persona Management
- Survey Simulation Engine
- LLM Inference orchestration
- Analytics Engine
- Data files (glassdoor.json, product‑reviews.json, intel_product_reviews.json) serve as input corpora.
- Requirements, Dockerfile, fly.toml, and .env configuration indicate containerized deployment.

##### README Insights
- Architecture overview lists five subsystems:
1. Persona Management
2. Survey Simulation
3. LLM Inference
4. Analytics Engine
5. API Server
- External dependencies: OpenAI, Azure OpenAI, Anthropic, AWS for potential storage/logging.
- No persistent database; data is read from JSON or passed through memory.

##### Main Components & Relationships
a. Components:
- Client (Market Researcher or UI)
- API Server (FastAPI in server.py)
- Persona Management Module (personas.py, schema.py, SurveyTypes.py)
- Survey Simulation Engine (survey_simulation.py, question_classifier.py)
- Ask Endpoint Module (ask_endpoint/)
- Deep Research Engine (deep_research/)
- Meta‑Analysis Module (survey_meta_analysis/)
- Analytics Engine (response_analytics.py, qualitative_analytics.py)
- Data Sources (static JSON files)
- LLM Providers (OpenAI/Azure/Anthropic – llminference.py, prompts.py)
- Configuration and Deployment (Docker, fly.toml, .env)
b. Interactions:
1. Client → API Server: HTTP POST /ask, /survey/run, /research
2. API Server routes request to appropriate module:
– /ask → ask_endpoint
– /survey/run → Survey Simulation Engine
– /research → Deep Research Engine
3. Within Simulation or Ask modules:
– Persona Management loads persona profile from JSON or in‑memory schema.
– Orchestrator distributes questions to multiple persona instances (parallel/concurrent).
– For each question, call LLM Inference → external LLM services → receive response.
4. Collected responses → Analytics Engine (response_analytics, qualitative_analytics, network & sentiment analysis).
5. Meta‑Analysis Module may consume analytics results for higher‑level insights.
6. Final JSON payload sent back to Client.
c. Architectural Patterns:
- Modular monolith: distinct folders for each domain.
- Pipeline pattern in deep_research (waterfall of prompt → LLM call → analysis → repeat).
- Parallel/concurrent processing of persona simulation.
- Adapter pattern for LLM providers (llminference.py abstracts OpenAI, Azure, Anthropic).
d. Key Technologies:
- Python 3.8+
- FastAPI for REST endpoints
- Asyncio or concurrent.futures for parallel simulations
- Azure/OpenAI/Anthropic APIs
- Docker + Fly.io for containerized deployment
- JSON files as data stores

## Diagram (Mermaid)

```mermaid
flowchart TD
    Client["Client (Researcher/UI)"]:::client
    API["API Server\n(server.py)"]:::api

    subgraph "API Endpoints"
        AskEP["Ask Endpoint\n(ask_endpoint/)"]:::core
        SurveyEP["Survey Endpoint\n(/survey/run)"]:::core
        ResearchEP["Research Endpoint\n(/research)"]:::core
    end

    subgraph "Core Subsystems"
        PersonaMgmt["Persona Management\n(personas.py,\nschema.py,\nSurveyTypes.py,\ntransform_schema.py)"]:::core
        Simulation["Survey Simulation Engine\n(survey_simulation.py,\nquestion_classifier.py,\nsurvey_status.py)"]:::core
        DeepResearch["Deep Research Engine\n(deep_research/)"]:::core
        Analytics["Analytics Engine\n(response_analytics.py,\nqualitative_analytics.py)"]:::core
        MetaAnalysis["Meta-Analysis Module\n(survey_meta_analysis/,\nsurvey_meta_analysis.py)"]:::core
        LLMLayer["LLM Inference Layer\n(llminference.py,\nprompts.py,\nclaude.py,\ngemini-request.py)"]:::core
    end

    subgraph "Data Sources"
        DataFiles["Static JSON Files\n(glassdoor.json,\nproduct-reviews.json,\nintel_product_reviews.json)"]:::data
    end

    subgraph "External LLM Providers"
        OpenAI["OpenAI"]:::external
        Azure["Azure OpenAI"]:::external
        Anthropic["Anthropic"]:::external
    end

    subgraph "Deployment & Config"
        Deployment["Config & Deployment\n(Dockerfile,\nfly.toml,\nrequirements.txt,\n.dockerignore,\n.gitignore)"]:::deployment
    end

    Client -->|"HTTP POST /ask\n/survey/run\n/research"| API
    API --> AskEP
    API --> SurveyEP
    API --> ResearchEP

    AskEP --> PersonaMgmt
    SurveyEP --> PersonaMgmt
    ResearchEP --> DeepResearch

    PersonaMgmt --> DataFiles
    SurveyEP --> Simulation
    Simulation -->|"Parallel questions\nacross N personas"| PersonaMgmt

    AskEP --> LLMLayer
    Simulation --> LLMLayer
    DeepResearch --> LLMLayer

    LLMLayer --> OpenAI
    LLMLayer --> Azure
    LLMLayer --> Anthropic

    AskEP --> Analytics
    Simulation --> Analytics
    DeepResearch --> Analytics

    Analytics --> MetaAnalysis
    MetaAnalysis --> API

    Deployment --- API

    click API "https://github.com/aliyanishfaq/ai-market-researcher/blob/main/server.py"
    click AskEP "https://github.com/aliyanishfaq/ai-market-researcher/tree/main/ask_endpoint/"
    click PersonaMgmt "https://github.com/aliyanishfaq/ai-market-researcher/blob/main/personas.py"
    click Simulation "https://github.com/aliyanishfaq/ai-market-researcher/blob/main/survey_simulation.py"
    click DeepResearch "https://github.com/aliyanishfaq/ai-market-researcher/tree/main/deep_research/"
    click MetaAnalysis "https://github.com/aliyanishfaq/ai-market-researcher/tree/main/survey_meta_analysis/"
    click Analytics "https://github.com/aliyanishfaq/ai-market-researcher/blob/main/response_analytics.py"
    click LLMLayer "https://github.com/aliyanishfaq/ai-market-researcher/blob/main/llminference.py"
    click DataFiles "https://github.com/aliyanishfaq/ai-market-researcher/blob/main/glassdoor.json"

    classDef client fill:#AEDFF7,stroke:#000
    classDef api fill:#ADD8E6,stroke:#000
    classDef core fill:#C6F5C6,stroke:#000
    classDef external fill:#E3C6F5,stroke:#000
    classDef data fill:#C6A46C,stroke:#000
    classDef deployment fill:#DADADA,stroke:#000
```
