How impredicative/newssurvey works
User Interface Layer
• Command-line interface
• Library interface
Core Workflow Controller
• 8-step workflow process described in the README
Source Adapters Layer
• Search and article components for each source
• Connection to external news sites
LLM Integration Layer
• OpenAI API integration (gpt-4o-mini and gpt-4o)
• Prompt management system
• Token counting and context management
Caching System
• Disk cache component
• Separate caches for website and LLM outputs
• Expiration management
Utility Services
• Text processing utilities
• Threading management
• Configuration management
[User Interface Layer]
• ↓
[Workflow Controller]
• ↙ ↓ ↘
[Sources] [LLM] [Cache]
• ↓ ↓ ↓
[External Services Layer]
• Blue: User interface components
• Green: Core workflow components
• Yellow: External integrations (news sources, OpenAI)
• Purple: Caching and storage
• Gray: Utility services
Important Elements to Highlight:
Workflow Pipeline
• Data flow between steps
• Parallel processing capabilities
Data Flow Patterns
• Article retrieval flow
• Search term accumulation
• Section management
• Rating and filtering processes
Integration Points
• News source API connections
• OpenAI API integration
• Cache interaction points
Error Handling
• Retry mechanisms
• Error propagation paths
• Fallback strategies
Special Considerations:
Context Management
• How the 128K token limit is handled
• Batching mechanisms
• Article condensation flow
Caching Strategy
• Two-tier caching system
• Expiration periods
• Cache bypass mechanisms
Format Conversion
• Format transformation pipeline
• Supported output formats (txt, md, html, pdf, json)
Use standardized symbols:
• Rectangles for processing components
• Diamonds for decision points
• Clouds for external services
Annotations for:
• Token limits
• Cache expiration times
• API rate limits
• Maximum section constraints
This system design should effectively communicate both the high-level architecture and the important implementation details of this specialized news survey application.