# stochastic-sisyphus/synsearch architecture

> Architecture diagram of the GitHub repository stochastic-sisyphus/synsearch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/stochastic-sisyphus/synsearch
- Repository: https://github.com/stochastic-sisyphus/synsearch

## Overview

##### Main Components

a) Core Processing Pipeline:
- Data Ingestion Layer
- ArXiv API Integration
- Dataset Loaders (ScisummNet, XL-Sum)
- Custom Dataset Handlers

b) Processing Layers:
- Preprocessing Module
- Embedding Generation Module
- Clustering Module
- Summarization Module

c) Support Systems:
- Caching System
- Evaluation Pipeline
- Visualization Components
- API Layer

**[Data Sources] → [Processing Pipeline] → [Output/Visualization]**
- ↑               ↑                          ↑
- [Cache System]────────[Evaluation]──┘

##### Component Details

a) Data Processing Flow:
**[Data Sources] → [DataLoader] → [Preprocessor] → [EmbeddingGenerator] → [ClusterManager] → [Summarizer]**

b) Parallel Systems:
**[Cache Manager]   [Evaluation Pipeline]   [Visualization Tools]**
- ↓                   ↓                      ↓
**[Shared Resource Layer / Utils]**

##### Specific Implementation Details

a) Data Flow Components:
- src/data_loader/flexible_loader.py
- src/preprocessing/domain_agnostic_preprocessor.py
- src/embedding_generator.py
- src/clustering/cluster_manager.py
- src/summarization/hybrid_summarizer.py

b) Support Systems:
- src/utils/caching.py
- src/evaluation/eval_pipeline.py
- src/visualization/cluster_visualizer.py

##### Key Relationships to Emphasize

- Bidirectional flow between cache system and processing pipeline
- Parallel nature of evaluation metrics
- Demonstrate how the dashboard integrates with all components

##### Technical Stack Integration

Technology indicators:
- Python 3.8+
- CUDA/GPU acceleration
- BERT/T5 models
- HDBSCAN clustering

Create three distinct layers:

##### Performance Optimization Points

Areas where performance optimization occurs:
- Batch processing nodes
- Caching points
- Parallel processing capabilities
- GPU acceleration points

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Style Definitions
    classDef core fill:#2374f7,color:white
    classDef data fill:#4CAF50,color:white
    classDef support fill:#FFC107,color:black
    classDef critical fill:#f44336,color:white
    classDef utils fill:#9E9E9E,color:white

    %% Data Sources Layer
    subgraph DataSources
        ArxivAPI["ArXiv API"]:::data
        Datasets["Scientific Datasets"]:::data
        CustomData["Custom Data Sources"]:::data
    end

    %% Core Processing Pipeline
    subgraph ProcessingPipeline
        DataLoader["Dataset Loader"]:::core
        Preprocessor["Preprocessor"]:::core
        EmbeddingGen["Embedding Generator"]:::core
        ClusterMgr["Cluster Manager"]:::core
        Summarizer["Summarizer"]:::core
    end

    %% Support Systems
    subgraph SupportSystems
        Cache["Cache System"]:::support
        Evaluator["Evaluation Pipeline"]:::support
        Visualizer["Visualization System"]:::support
        ErrorHandler["Error Handler"]:::utils
        PerfMonitor["Performance Monitor"]:::utils
    end

    %% User Interface
    Dashboard["Dashboard Interface"]:::core

    %% Relationships
    ArxivAPI --> DataLoader
    Datasets --> DataLoader
    CustomData --> DataLoader
    
    DataLoader --> Preprocessor
    Preprocessor --> EmbeddingGen
    EmbeddingGen --> ClusterMgr
    ClusterMgr --> Summarizer
    
    Cache --> ProcessingPipeline
    ProcessingPipeline --> Cache
    
    Summarizer --> Dashboard
    Visualizer --> Dashboard
    
    ProcessingPipeline --> Evaluator
    Evaluator --> Dashboard
    
    ErrorHandler --> ProcessingPipeline
    PerfMonitor --> ProcessingPipeline

    %% Click Events
    click ArxivAPI "https://github.com/stochastic-sisyphus/synsearch/blob/main/src/api/arxiv_api.py"
    click DataLoader "https://github.com/stochastic-sisyphus/synsearch/blob/main/src/data_loader/flexible_loader.py"
    click Preprocessor "https://github.com/stochastic-sisyphus/synsearch/blob/main/src/preprocessing/domain_agnostic_preprocessor.py"
    click EmbeddingGen "https://github.com/stochastic-sisyphus/synsearch/blob/main/src/embedding_generator.py"
    click ClusterMgr "https://github.com/stochastic-sisyphus/synsearch/blob/main/src/clustering/cluster_manager.py"
    click Summarizer "https://github.com/stochastic-sisyphus/synsearch/blob/main/src/summarization/hybrid_summarizer.py"
    click Cache "https://github.com/stochastic-sisyphus/synsearch/blob/main/src/utils/caching.py"
    click Evaluator "https://github.com/stochastic-sisyphus/synsearch/blob/main/src/evaluation/eval_pipeline.py"
    click Visualizer "https://github.com/stochastic-sisyphus/synsearch/blob/main/src/visualization/cluster_visualizer.py"
    click Dashboard "https://github.com/stochastic-sisyphus/synsearch/blob/main/src/dashboard/app.py"
    click ErrorHandler "https://github.com/stochastic-sisyphus/synsearch/blob/main/src/utils/error_handler.py"
    click PerfMonitor "https://github.com/stochastic-sisyphus/synsearch/blob/main/src/utils/performance.py"

    %% Legend
    subgraph Legend
        L1["Core Components"]:::core
        L2["Data Sources"]:::data
        L3["Support Systems"]:::support
        L4["Utils"]:::utils
    end
```
