# usharma123/startup-success-forecasting-framework architecture

> Architecture diagram of the GitHub repository usharma123/startup-success-forecasting-framework, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/usharma123/startup-success-forecasting-framework
- Repository: https://github.com/usharma123/startup-success-forecasting-framework

## Overview

##### Project Type and Purpose
- SSFF is a Python-based machine-learning pipeline with an interactive Streamlit UI.
- It ingests real-time market data and startup metadata, applies preprocessing, trains or loads models (ML and LLM), and surfaces success-forecast scores.
- Key stakeholders: data scientists (EDA/experiments), ML engineers (models/algorithms), and end users (Streamlit app).

##### Main Components
1. Data Sources (External Services)
– SerpAPI (real-time market data)
– OpenAI API (LLM embeddings/analysis)
– Potential future sources: Crunchbase, LinkedIn, Twitter
2. Utility & Configuration
– utils/config.py (env management, API keys)
– utils/api_wrapper.py (wrappers for SerpAPI & OpenAI)
3. Data Ingestion & Preprocessing
– EDA notebooks (exploratory analysis, scripts in EDA/)
– algorithms/preprocessing.py (clean, encode, feature extraction)
4. Feature Engineering & Algorithms
– algorithms/embedding.py (LLM embeddings)
– algorithms/similarity.py (market‐fit metrics)
– algorithms/LLM_Segmentation.py, randomForest.py, neuralNetworks.py
5. Model Management
– models/… (trained_encoder_RF.joblib, random_forest_classifier.joblib, neural_network.keras, etc.)
– overallPipeline.py, baseline_framework.py, ssff_framework.py (pipeline orchestration)
6. Agents Layer
– agents/base_agent.py (abstract interface)
– agents/market_agent.py, founder_agent.py, product_agent.py, vc_scout_agent.py, integration_agent.py
- Encapsulate domain-specific logic and calls to pipelines or APIs
7. Experiments
– experiments/ (notebooks and scripts for new ideas and benchmarking)
8. User Interface / API
– app.py (Streamlit frontend)
– main.py or CLI entrypoints if present
9. CI/CD & Deployment
– .github/workflows/pylint.yml (linting)
– requirements.txt, .env for config

##### Relationships & Data Flow
- External Services → utils/api_wrapper → Data Ingestion
- Data Ingestion → algorithms/preprocessing → feature store (in-memory or files)
- Features → Feature Engineering modules → Model Manager (load/train/serialize)
- Models → Agents (each agent invokes pipelines/models to produce domain outputs)
- Agents → integration_agent → final scoring pipeline
- Scoring Pipeline → app.py (Streamlit) → End User

##### Architectural Patterns & Principles
- Pipeline Pattern: clear stages (ingest → preprocess → feature → model → serve)
- Modularization: separate directories for EDA, algorithms, models, agents, utils
- Agent-Oriented Design: encapsulate domain tasks in agent classes adhering to a common interface
- Extensibility: add new data sources or agents by dropping in new modules under utils or agents
- Configuration-Driven: all secrets and endpoints via config.py/.env

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Services
    subgraph "External Data & APIs"
        SerpAPI["SerpAPI"]:::external
        OpenAI["OpenAI API"]:::external
    end

    %% Utils & Config
    subgraph "Utils & Configuration"
        Config["Configuration (config.py)"]:::pipelines
        APIWrapper["Data Sources Wrapper (api_wrapper.py)"]:::pipelines
    end

    %% Ingestion & Preprocessing
    subgraph "Ingestion & Preprocessing"
        EDA["Exploratory Data Analysis (EDA/)"]:::pipelines
        Preprocessing["Preprocessing Module"]:::pipelines
    end

    %% Feature Engineering & Algorithms
    subgraph "Feature Engineering & Algorithms"
        Embedding["Embedding Module"]:::pipelines
        Similarity["Similarity Module"]:::pipelines
        LLMSeg["LLM Segmentation (LLM_Segmentation.py)"]:::pipelines
        RFAlg["Random Forest (randomForest.py)"]:::pipelines
        NN["Neural Networks (neuralNetworks.py)"]:::pipelines
    end

    %% Model Management
    subgraph "Model Management"
        OverallPipeline["Pipeline Orchestration (overallPipeline.py)"]:::pipelines
        Baseline["Baseline Framework (baseline_framework.py)"]:::pipelines
        SSFFPipeline["SSFF Framework (ssff_framework.py)"]:::pipelines
        ModelsStore["Model Store (models/)"]:::models
    end

    %% Agents Layer
    subgraph "Agents Layer"
        BaseAgent["Base Agent (base_agent.py)"]:::agents
        MarketAgent["Market Agent (market_agent.py)"]:::agents
        FounderAgent["Founder Agent (founder_agent.py)"]:::agents
        ProductAgent["Product Agent (product_agent.py)"]:::agents
        VCScout["VC Scout Agent (vc_scout_agent.py)"]:::agents
        IntegrationAgent["Integration Agent (integration_agent.py)"]:::agents
    end

    %% User Interface
    UI["Streamlit UI (app.py)"]:::ui

    %% Experiments
    subgraph "Experiments"
        Experiments["Experiments (experiments/)"]:::experiments
    end

    %% CI/CD
    subgraph "CI/CD & Deployment"
        CICD["CI/CD Workflow (pylint.yml)"]:::cicd
        Requirements["Dependencies (requirements.txt)"]:::cicd
    end

    %% Data Flow
    SerpAPI -->|calls| APIWrapper
    OpenAI -->|calls| APIWrapper
    Config -->|provides| APIWrapper

    APIWrapper --> Preprocessing
    EDA --> Preprocessing

    Preprocessing --> Embedding
    Preprocessing --> Similarity

    Embedding --> OverallPipeline
    Similarity --> OverallPipeline
    LLMSeg --> OverallPipeline
    RFAlg --> OverallPipeline
    NN --> OverallPipeline

    OverallPipeline --> ModelsStore
    Baseline --> ModelsStore
    SSFFPipeline --> ModelsStore

    ModelsStore --> MarketAgent
    ModelsStore --> FounderAgent
    ModelsStore --> ProductAgent
    ModelsStore --> VCScout

    MarketAgent --> IntegrationAgent
    FounderAgent --> IntegrationAgent
    ProductAgent --> IntegrationAgent
    VCScout --> IntegrationAgent

    IntegrationAgent --> SSFFPipeline
    SSFFPipeline --> UI

    Experiments -.-> Preprocessing
    Experiments -.-> Embedding
    Experiments -.-> RFAlg
    Experiments -.-> ModelsStore

    CICD --> Config
    CICD --> APIWrapper
    CICD --> Preprocessing
    CICD --> OverallPipeline
    CICD --> BaseAgent
    CICD --> UI

    %% Click Events
    click APIWrapper "https://github.com/usharma123/startup-success-forecasting-framework/blob/main/utils/api_wrapper.py"
    click Config "https://github.com/usharma123/startup-success-forecasting-framework/blob/main/utils/config.py"
    click EDA "https://github.com/usharma123/startup-success-forecasting-framework/tree/main/EDA/"
    click LLMSeg "https://github.com/usharma123/startup-success-forecasting-framework/blob/main/algorithms/LLM_Segmentation.py"
    click RFAlg "https://github.com/usharma123/startup-success-forecasting-framework/blob/main/algorithms/randomForest.py"
    click NN "https://github.com/usharma123/startup-success-forecasting-framework/blob/main/algorithms/neuralNetworks.py"
    click OverallPipeline "https://github.com/usharma123/startup-success-forecasting-framework/blob/main/overallPipeline.py"
    click Baseline "https://github.com/usharma123/startup-success-forecasting-framework/blob/main/baseline_framework.py"
    click SSFFPipeline "https://github.com/usharma123/startup-success-forecasting-framework/blob/main/ssff_framework.py"
    click BaseAgent "https://github.com/usharma123/startup-success-forecasting-framework/blob/main/agents/base_agent.py"
    click MarketAgent "https://github.com/usharma123/startup-success-forecasting-framework/blob/main/agents/market_agent.py"
    click FounderAgent "https://github.com/usharma123/startup-success-forecasting-framework/blob/main/agents/founder_agent.py"
    click ProductAgent "https://github.com/usharma123/startup-success-forecasting-framework/blob/main/agents/product_agent.py"
    click VCScout "https://github.com/usharma123/startup-success-forecasting-framework/blob/main/agents/vc_scout_agent.py"
    click IntegrationAgent "https://github.com/usharma123/startup-success-forecasting-framework/blob/main/agents/integration_agent.py"
    click ModelsStore "https://github.com/usharma123/startup-success-forecasting-framework/tree/main/models/"
    click Experiments "https://github.com/usharma123/startup-success-forecasting-framework/tree/main/experiments/"
    click UI "https://github.com/usharma123/startup-success-forecasting-framework/blob/main/app.py"
    click CICD "https://github.com/usharma123/startup-success-forecasting-framework/blob/main/.github/workflows/pylint.yml"
    click Requirements "https://github.com/usharma123/startup-success-forecasting-framework/blob/main/requirements.txt"

    %% Styles
    classDef external fill:#ADD8E6,stroke:#333
    classDef pipelines fill:#98FB98,stroke:#333
    classDef models fill:#FFFF99,stroke:#333
    classDef agents fill:#DDA0DD,stroke:#333
    classDef ui fill:#FFA500,stroke:#333
    classDef experiments fill:#FFD700,stroke:#333
    classDef cicd fill:#D3D3D3,stroke:#333
```
