# impredicative/newssurvey architecture

> Architecture diagram of the GitHub repository impredicative/newssurvey, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/impredicative/newssurvey
- Repository: https://github.com/impredicative/newssurvey

## Overview

##### User Interface Layer
- Command-line interface
- Library interface

##### Core Workflow Controller
- 8-step workflow process described in the README

##### Source Adapters Layer
- Search and article components for each source
- Connection to external news sites

##### LLM Integration Layer
- OpenAI API integration (gpt-4o-mini and gpt-4o)
- Prompt management system
- Token counting and context management

##### Caching System
- Disk cache component
- Separate caches for website and LLM outputs
- Expiration management

##### Utility Services
- Text processing utilities
- Threading management
- Configuration management

**[User Interface Layer]**
- ↓
**[Workflow Controller]**
- ↙     ↓     ↘
**[Sources] [LLM] [Cache]**
- ↓       ↓      ↓
**[External Services Layer]**

- Blue: User interface components
- Green: Core workflow components
- Yellow: External integrations (news sources, OpenAI)
- Purple: Caching and storage
- Gray: Utility services

Important Elements to Highlight:

##### Workflow Pipeline
- Data flow between steps
- Parallel processing capabilities

##### Data Flow Patterns
- Article retrieval flow
- Search term accumulation
- Section management
- Rating and filtering processes

##### Integration Points
- News source API connections
- OpenAI API integration
- Cache interaction points

##### Error Handling
- Retry mechanisms
- Error propagation paths
- Fallback strategies

Special Considerations:

##### Context Management
- How the 128K token limit is handled
- Batching mechanisms
- Article condensation flow

##### Caching Strategy
- Two-tier caching system
- Expiration periods
- Cache bypass mechanisms

##### Format Conversion
- Format transformation pipeline
- Supported output formats (txt, md, html, pdf, json)

Use standardized symbols:
- Rectangles for processing components
- Diamonds for decision points
- Clouds for external services

Annotations for:
- Token limits
- Cache expiration times
- API rate limits
- Maximum section constraints

This system design should effectively communicate both the high-level architecture and the important implementation details of this specialized news survey application.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Styles and Classes
    classDef ui fill:#FFE5CC,stroke:#FF9933
    classDef workflow fill:#E6E6FF,stroke:#6666FF
    classDef source fill:#CCE5CC,stroke:#339933
    classDef llm fill:#FFE6FF,stroke:#993399
    classDef storage fill:#FFE6E6,stroke:#993333
    classDef external fill:#E6E6E6,stroke:#666666

    %% User Interface Layer
    subgraph UI["User Interface Layer"]
        CLI["CLI Interface"]:::ui
        Config["Configuration Handler"]:::ui
        EnvConfig["Environment Config"]:::ui
    end

    %% Workflow Orchestration Layer
    subgraph Workflow["Workflow Orchestration"]
        LLMWorkflow["LLM Workflow"]:::workflow
        SourceWorkflow["Source Workflow"]:::workflow
        UserWorkflow["User Workflow"]:::workflow
    end

    %% Source Integration Layer
    subgraph Sources["Source Integration"]
        MedXpress["MedicalXpress Connector"]:::source
        PhysOrg["PhysOrg Connector"]:::source
        SourceUtils["Common Source Utilities"]:::source
    end

    %% LLM Processing Layer
    subgraph LLM["LLM Processing"]
        OpenAI["OpenAI API Integration"]:::llm
        Prompts["Prompt Management"]:::llm
        TokenMgmt["Token Management"]:::llm
    end

    %% Storage Layer
    subgraph Storage["Storage Layer"]
        Cache["Disk Cache System"]:::storage
        Output["Output Formatter"]:::storage
    end

    %% External Services
    ExternalOpenAI["OpenAI API"]:::external
    ExternalNews["News Sources"]:::external

    %% Core Components
    Types["Type Definitions"]:::workflow
    Exceptions["Exception Handler"]:::workflow
    MainApp["Main Application"]:::workflow

    %% Relationships
    CLI --> UserWorkflow
    Config --> MainApp
    EnvConfig --> Config
    
    UserWorkflow --> SourceWorkflow
    UserWorkflow --> LLMWorkflow
    SourceWorkflow --> Sources
    LLMWorkflow --> LLM
    
    MedXpress --> ExternalNews
    PhysOrg --> ExternalNews
    OpenAI --> ExternalOpenAI
    
    Sources --> Cache
    LLM --> Cache
    Cache --> Output
    Output --> CLI

    %% Click Events
    click CLI "https://github.com/impredicative/newssurvey/blob/master/src/newssurvey/__main__.py"
    click Config "https://github.com/impredicative/newssurvey/blob/master/src/newssurvey/config.py"
    click EnvConfig "https://github.com/impredicative/newssurvey/blob/master/src/newssurvey/util/dotenv_.py"
    click LLMWorkflow "https://github.com/impredicative/newssurvey/tree/master/src/newssurvey/workflow/llm/"
    click SourceWorkflow "https://github.com/impredicative/newssurvey/tree/master/src/newssurvey/workflow/source/"
    click UserWorkflow "https://github.com/impredicative/newssurvey/tree/master/src/newssurvey/workflow/user/"
    click MedXpress "https://github.com/impredicative/newssurvey/tree/master/src/newssurvey/sources/medicalxpress/"
    click PhysOrg "https://github.com/impredicative/newssurvey/tree/master/src/newssurvey/sources/physorg/"
    click SourceUtils "https://github.com/impredicative/newssurvey/blob/master/src/newssurvey/sources/medicalxpress/_common.py"
    click OpenAI "https://github.com/impredicative/newssurvey/blob/master/src/newssurvey/util/openai_.py"
    click Prompts "https://github.com/impredicative/newssurvey/tree/master/src/newssurvey/prompts/"
    click TokenMgmt "https://github.com/impredicative/newssurvey/blob/master/src/newssurvey/util/tiktoken_.py"
    click Cache "https://github.com/impredicative/newssurvey/blob/master/src/newssurvey/util/diskcache_.py"
    click Output "https://github.com/impredicative/newssurvey/blob/master/src/newssurvey/workflow/user/output.py"
    click Types "https://github.com/impredicative/newssurvey/blob/master/src/newssurvey/types.py"
    click Exceptions "https://github.com/impredicative/newssurvey/blob/master/src/newssurvey/exceptions.py"
    click MainApp "https://github.com/impredicative/newssurvey/blob/master/src/newssurvey/newssurvey.py"
```
