How usharma123/startup-success-forecasting-framework works
Project Type and Purpose
• SSFF is a Python-based machine-learning pipeline with an interactive Streamlit UI.
• It ingests real-time market data and startup metadata, applies preprocessing, trains or loads models (ML and LLM), and surfaces success-forecast scores.
• Key stakeholders: data scientists (EDA/experiments), ML engineers (models/algorithms), and end users (Streamlit app).
Main Components
1. Data Sources (External Services)
– SerpAPI (real-time market data)
– OpenAI API (LLM embeddings/analysis)
– Potential future sources: Crunchbase, LinkedIn, Twitter
2. Utility & Configuration
– utils/config.py (env management, API keys)
– utils/api_wrapper.py (wrappers for SerpAPI & OpenAI)
3. Data Ingestion & Preprocessing
– EDA notebooks (exploratory analysis, scripts in EDA/)
– algorithms/preprocessing.py (clean, encode, feature extraction)
4. Feature Engineering & Algorithms
– algorithms/embedding.py (LLM embeddings)
– algorithms/similarity.py (market‐fit metrics)
– algorithms/LLM_Segmentation.py, randomForest.py, neuralNetworks.py
5. Model Management
– models/… (trained_encoder_RF.joblib, random_forest_classifier.joblib, neural_network.keras, etc.)
– overallPipeline.py, baseline_framework.py, ssff_framework.py (pipeline orchestration)
6. Agents Layer
– agents/base_agent.py (abstract interface)
– agents/market_agent.py, founder_agent.py, product_agent.py, vc_scout_agent.py, integration_agent.py
• Encapsulate domain-specific logic and calls to pipelines or APIs
7. Experiments
– experiments/ (notebooks and scripts for new ideas and benchmarking)
8. User Interface / API
– app.py (Streamlit frontend)
– main.py or CLI entrypoints if present
9. CI/CD & Deployment
– .github/workflows/pylint.yml (linting)
– requirements.txt, .env for config
Relationships & Data Flow
• External Services → utils/api_wrapper → Data Ingestion
• Data Ingestion → algorithms/preprocessing → feature store (in-memory or files)
• Features → Feature Engineering modules → Model Manager (load/train/serialize)
• Models → Agents (each agent invokes pipelines/models to produce domain outputs)
• Agents → integration_agent → final scoring pipeline
• Scoring Pipeline → app.py (Streamlit) → End User
Architectural Patterns & Principles
• Pipeline Pattern: clear stages (ingest → preprocess → feature → model → serve)
• Modularization: separate directories for EDA, algorithms, models, agents, utils
• Agent-Oriented Design: encapsulate domain tasks in agent classes adhering to a common interface
• Extensibility: add new data sources or agents by dropping in new modules under utils or agents
• Configuration-Driven: all secrets and endpoints via config.py/.env