# madhavendrapandey/trend_intelligence_system architecture

> Architecture diagram of the GitHub repository madhavendrapandey/trend_intelligence_system, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/madhavendrapandey/trend_intelligence_system
- Repository: https://github.com/madhavendrapandey/trend_intelligence_system

## Overview

1. **Repository shape**
   - This is a batch-oriented intelligence platform, not a single app. The core flow is: external sources are collected, filtered and analyzed, then persisted and rendered into reports and a web UI.
   - The codebase is split into two main intelligence domains: **trend intelligence** and **friction intelligence**.

2. **External data ingestion**
   - The `core/collectors` layer pulls raw content from multiple sources: **RSS, GitHub, arXiv, Hacker News, and Reddit**.
   - Source-specific configuration lives in `core/sources`, which acts as the source registry / selectors for what gets collected.

3. **Pre-processing and signal hygiene**
   - Before analysis, items pass through filters in `core/filters`:
     - duplicate detection
     - content quality checks
     - source quality checks
     - interest scoring / relevance filtering
     - Reddit-specific quality filtering
   - This is the first major boundary: raw internet noise is reduced to a smaller candidate set for intelligence work.

4. **LLM abstraction**
   - `core/llm/providers.py` centralizes model-provider access and supports both local and API-based LLM backends.
   - The README indicates support for Ollama/Qwen locally, plus cloud providers like Gemini, OpenAI, Anthropic, Groq, OpenRouter, and Together AI.
   - This layer is used by downstream extraction/grouping services rather than being coupled directly to the UI or collectors.

5. **Domain models**
   - `core/models` defines the primary business objects: evidence, evidence groups, friction candidates, friction profiles, validation events, and related Qwen/model abstractions.
   - These models make the analysis pipeline explicit: raw items become evidence, evidence is grouped, grouped evidence informs candidates/profiles, and validation tracks confidence or review state.

6. **Trend intelligence pipeline**
   - `modules/trend` is the trend-analysis subsystem.
   - Its engines handle:
     - signal extraction
     - topic normalization
     - trend scoring
     - opportunity detection
     - recommendation generation
   - `modules/trend/services/trend_intelligence_service.py` likely orchestrates these engines into a coherent trend report pipeline.

7. **Friction intelligence pipeline**
   - `modules/friction` is a separate analysis stack focused on friction, contradictions, and validation.
   - It includes:
     - evidence extraction
     - evidence grouping
     - contradiction analysis
     - candidate generation
     - relationship/evolution/profile services
     - traceability and validation services
   - This subsystem appears designed to turn collected observations into structured “friction intelligence” objects rather than generic trend summaries.

8. **Persistence and schema evolution**
   - `database/` contains the SQLite-backed persistence layer: repositories for posts, sources, runs, evidence, groups, candidates, profiles, validations, trends, operators, and failure tracking.
   - Migrations show the system evolved from a simpler article store to a richer schema with evidence, friction, operator tooling, and trend intelligence.
   - The presence of `SQLITE_CUTOVER_REPORT.md` and migration scripts suggests a deliberate transition from JSON storage to SQLite as the authoritative store.

9. **Storage boundary and repositories**
   - `core/storage` contains storage interfaces plus JSON and SQLite implementations, so the app can work across persistence backends.
   - `database/repositories` is the main data-access layer used by higher-level services and the website.
   - This cleanly separates analysis logic from storage concerns.

10. **Reporting outputs**
   - Reports are generated into `reports/`, including daily markdown/JSON artifacts and weekly briefs.
   - Top-level scripts like `reporter.py`, `analyzer.py`, and `main.py` indicate a staged pipeline where collection, analysis, and report generation can run independently or end-to-end.
   - There are also HTML outputs such as `evidence_output.html` and `operator_output.html`, suggesting artifact-based inspection is part of the workflow.

11. **Web application and operator console**
   - `website/` is a presentation layer over the stored intelligence.
   - It uses routes, templates, static assets, and service abstractions to render pages for evidence, sources, profiles, reports, and operator workflows.
   - The operator section includes dashboard, explorer, pipeline monitor, SQL studio, schema browser, and table browser views, which makes the persistence layer inspectable for debugging and quality control.

12. **Operational tooling and metrics**
   - `scheduler.py` suggests automated periodic runs.
   - `stats/` tracks collection statistics, which likely feed monitoring or quality reporting.
   - `database/compare_report_parity.py` and migration utilities indicate the system includes tooling for validating the new database-backed pipeline against prior report outputs.

13. **Primary end-to-end data flow**
   - External sources → collectors → filters/deduplication → extraction/LLM analysis → trend/friction engines → SQLite repositories → reports and website.
   - The most important architectural boundaries are: ingestion vs analysis, analysis vs persistence, and persistence vs presentation.

14. **What to draw in an architecture diagram**
   - Show two parallel intelligence lanes sharing the same ingestion and storage foundation: **Trend Intelligence** and **Friction Intelligence**.
   - Put the database/repository layer underneath both, with the website and reporting tools reading from it.
   - Highlight the LLM provider boundary as a shared dependency used selectively by analysis services, not by collectors or UI code.

Main entry points: [`database/repositories`](https://github.com/madhavendrapandey/trend_intelligence_system/tree/main/database/repositories) (Repositories), [`modules/friction/extractors`](https://github.com/madhavendrapandey/trend_intelligence_system/tree/main/modules/friction/extractors) (Evidence Extractors), [`modules/friction/services`](https://github.com/madhavendrapandey/trend_intelligence_system/tree/main/modules/friction/services) (Friction Services), [`core/filters`](https://github.com/madhavendrapandey/trend_intelligence_system/tree/main/core/filters) (Filters)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["Ingestion"]
  node_sources["Sources<br/>registry"]
  node_collectors["Collectors<br/>ingest"]
  node_filters["Filters<br/>hygiene"]
end

subgraph group_analysis["Analysis"]
  node_llm["LLM Providers<br/>model boundary<br/>[providers.py]"]
  node_models["Domain Models<br/>entities"]
  node_trend_service["Trend Service<br/>orchestrator"]
  node_trend_engines["Trend Engines<br/>pipeline"]
  node_friction_runner["Friction Runner<br/>orchestrator<br/>[runner.py]"]
  node_friction_extractors["Evidence Extractors<br/>extraction"]
  node_friction_services["Friction Services<br/>analysis stack"]
end

subgraph group_persistence["Persistence"]
  node_sqlite_storage[("SQLite Storage<br/>[sqlite_storage.py]")]
  node_json_storage[("JSON Storage<br/>[json_storage.py]")]
  node_repositories["Repositories<br/>data access"]
  node_migrations["Migrations<br/>schema evolution"]
end

subgraph group_presentation["Presentation"]
  node_reports["Reports<br/>artifacts"]
  node_reporter["Reporter<br/>batch job<br/>[reporter.py]"]
  node_website["Website<br/>web app"]
end

subgraph group_ops["Operations"]
  node_scheduler["Scheduler<br/>automation<br/>[scheduler.py]"]
  node_stats["Stats<br/>monitoring"]
end

node_sources -->|"selects"| node_collectors
node_collectors -->|"raw items"| node_filters
node_filters -->|"candidate items"| node_friction_extractors
node_filters -->|"signals"| node_trend_service
node_llm -->|"assist"| node_friction_extractors
node_llm -->|"assist"| node_friction_services
node_models -->|"shape evidence"| node_friction_extractors
node_models -->|"govern analysis"| node_friction_services
node_trend_service -->|"orchestrates"| node_trend_engines
node_friction_extractors -->|"feeds"| node_friction_services
node_friction_services -->|"persist"| node_repositories
node_trend_engines -->|"persist"| node_repositories
node_sqlite_storage -->|"backing store"| node_repositories
node_json_storage -.->|"legacy store"| node_repositories
node_migrations -->|"evolves schema"| node_repositories
node_repositories -->|"source data"| node_reports
node_repositories -->|"serve data"| node_website
node_reporter -->|"generates"| node_reports
node_scheduler -.->|"triggers"| node_collectors
node_scheduler -.->|"triggers"| node_reporter
node_stats -.->|"observes"| node_scheduler
node_website -->|"presents"| node_reporter

click node_sources "https://github.com/madhavendrapandey/trend_intelligence_system/tree/main/core/sources"
click node_collectors "https://github.com/madhavendrapandey/trend_intelligence_system/tree/main/core/collectors"
click node_filters "https://github.com/madhavendrapandey/trend_intelligence_system/tree/main/core/filters"
click node_llm "https://github.com/madhavendrapandey/trend_intelligence_system/blob/main/core/llm/providers.py"
click node_models "https://github.com/madhavendrapandey/trend_intelligence_system/tree/main/core/models"
click node_trend_service "https://github.com/madhavendrapandey/trend_intelligence_system/blob/main/modules/trend/services/trend_intelligence_service.py"
click node_trend_engines "https://github.com/madhavendrapandey/trend_intelligence_system/tree/main/modules/trend/engines"
click node_friction_runner "https://github.com/madhavendrapandey/trend_intelligence_system/blob/main/modules/friction/runner.py"
click node_friction_extractors "https://github.com/madhavendrapandey/trend_intelligence_system/tree/main/modules/friction/extractors"
click node_friction_services "https://github.com/madhavendrapandey/trend_intelligence_system/tree/main/modules/friction/services"
click node_sqlite_storage "https://github.com/madhavendrapandey/trend_intelligence_system/blob/main/core/storage/sqlite_storage.py"
click node_json_storage "https://github.com/madhavendrapandey/trend_intelligence_system/blob/main/core/storage/json_storage.py"
click node_repositories "https://github.com/madhavendrapandey/trend_intelligence_system/tree/main/database/repositories"
click node_migrations "https://github.com/madhavendrapandey/trend_intelligence_system/tree/main/database/migrations"
click node_reports "https://github.com/madhavendrapandey/trend_intelligence_system/tree/main/reports"
click node_reporter "https://github.com/madhavendrapandey/trend_intelligence_system/blob/main/reporter.py"
click node_website "https://github.com/madhavendrapandey/trend_intelligence_system/tree/main/website"
click node_scheduler "https://github.com/madhavendrapandey/trend_intelligence_system/blob/main/scheduler.py"
click node_stats "https://github.com/madhavendrapandey/trend_intelligence_system/tree/main/stats"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_sources,node_collectors,node_filters toneBlue
class node_llm,node_models,node_trend_service,node_trend_engines,node_friction_runner,node_friction_extractors,node_friction_services toneAmber
class node_sqlite_storage,node_json_storage,node_repositories,node_migrations toneMint
class node_reports,node_reporter,node_website toneRose
class node_scheduler,node_stats toneIndigo
```

## Components

### Ingestion

Source registry, collectors, and signal hygiene before analysis.

- [Sources](https://github.com/madhavendrapandey/trend_intelligence_system/tree/main/core/sources): registry · Source selectors and configuration for what gets collected. · `core/sources`
- [Collectors](https://github.com/madhavendrapandey/trend_intelligence_system/tree/main/core/collectors): ingest · Fetches raw content from RSS, GitHub, arXiv, Hacker News, and Reddit. · `core/collectors`
- [Filters](https://github.com/madhavendrapandey/trend_intelligence_system/tree/main/core/filters): hygiene · Duplicate, quality, source, interest, and Reddit-specific filtering. · `core/filters`

### Analysis

Parallel trend and friction intelligence pipelines.

- [LLM Providers](https://github.com/madhavendrapandey/trend_intelligence_system/blob/main/core/llm/providers.py): model boundary · Centralized access to local and hosted model backends. · `core/llm/providers.py`
- [Domain Models](https://github.com/madhavendrapandey/trend_intelligence_system/tree/main/core/models): entities · Evidence, groups, candidates, profiles, and validation events. · `core/models`
- [Trend Service](https://github.com/madhavendrapandey/trend_intelligence_system/blob/main/modules/trend/services/trend_intelligence_service.py): orchestrator · Coordinates the trend intelligence pipeline. · `modules/trend/services/trend_intelligence_service.py`
- [Trend Engines](https://github.com/madhavendrapandey/trend_intelligence_system/tree/main/modules/trend/engines): pipeline · Signal extraction, topic normalization, scoring, opportunity detection, and recommendations. · `modules/trend/engines`
- [Friction Runner](https://github.com/madhavendrapandey/trend_intelligence_system/blob/main/modules/friction/runner.py): orchestrator · Entry point for the friction intelligence workflow. · `modules/friction/runner.py`
- [Evidence Extractors](https://github.com/madhavendrapandey/trend_intelligence_system/tree/main/modules/friction/extractors): extraction · Builds evidence from collected material, including LLM-assisted extraction. · `modules/friction/extractors`
- [Friction Services](https://github.com/madhavendrapandey/trend_intelligence_system/tree/main/modules/friction/services): analysis stack · Grouping, contradiction, candidate, relationship, evolution, traceability, and validation services. · `modules/friction/services`

### Persistence

SQLite storage, repositories, and schema evolution.

- [SQLite Storage](https://github.com/madhavendrapandey/trend_intelligence_system/blob/main/core/storage/sqlite_storage.py): storage · Shared persistence interface implementation for SQLite. · `core/storage/sqlite_storage.py`
- [JSON Storage](https://github.com/madhavendrapandey/trend_intelligence_system/blob/main/core/storage/json_storage.py): storage · Legacy or alternate storage implementation behind the same interface. · `core/storage/json_storage.py`
- [Repositories](https://github.com/madhavendrapandey/trend_intelligence_system/tree/main/database/repositories): data access · Primary database repositories for evidence, trends, profiles, runs, and validation. · `database/repositories`
- [Migrations](https://github.com/madhavendrapandey/trend_intelligence_system/tree/main/database/migrations): schema evolution · Database schema and cutover support for evidence, friction, operator, and trend data. · `database/migrations`

### Presentation

Reports and web UI for browsing intelligence artifacts.

- [Reports](https://github.com/madhavendrapandey/trend_intelligence_system/tree/main/reports): artifacts · Daily and weekly report generation outputs. · `reports`
- [Reporter](https://github.com/madhavendrapandey/trend_intelligence_system/blob/main/reporter.py): batch job · Top-level report generation entry point. · `reporter.py`
- [Website](https://github.com/madhavendrapandey/trend_intelligence_system/tree/main/website): web app · Presentation layer for evidence, sources, profiles, reports, and operator workflows. · `website`

### Operations

Scheduling, stats, and migration tooling.

- [Scheduler](https://github.com/madhavendrapandey/trend_intelligence_system/blob/main/scheduler.py): automation · Periodic orchestration for collection and analysis runs. · `scheduler.py`
- [Stats](https://github.com/madhavendrapandey/trend_intelligence_system/tree/main/stats): monitoring · Collection statistics and operational metrics. · `stats`

## Connections

- Sources → Collectors: selects · Source registry determines which feeds and endpoints are collected.
- Collectors → Filters: raw items · Collected content is cleaned and reduced before analysis.
- Filters → Evidence Extractors: candidate items · Filtered items feed evidence extraction.
- Filters → Trend Service: signals · Clean items become inputs to trend intelligence.
- LLM Providers → Evidence Extractors: assist · LLM backends support evidence extraction where needed.
- LLM Providers → Friction Services: assist · LLM-backed grouping and analysis services use the shared provider layer.
- Domain Models → Evidence Extractors: shape evidence · Extractors produce evidence and related domain objects.
- Domain Models → Friction Services: govern analysis · Services operate on evidence groups, candidates, profiles, and validations.
- Trend Service → Trend Engines: orchestrates · Trend service composes the trend pipeline stages.
- Evidence Extractors → Friction Services: feeds · Extracted evidence flows into grouping, contradiction, and validation services.
- Friction Services → Repositories: persist · Friction intelligence artifacts are stored through repositories.
- Trend Engines → Repositories: persist · Trend outputs are written to the database layer.
- SQLite Storage → Repositories: backing store · Repositories read and write through the SQLite storage boundary.
- JSON Storage → Repositories: legacy store · Alternative storage path used by the storage interface.
- Migrations → Repositories: evolves schema · Schema changes support the repository layer as the data model grows.
- Repositories → Reports: source data · Reports are rendered from persisted intelligence objects.
- Repositories → Website: serve data · The web app reads persisted records for browsing and operator views.
- Reporter → Reports: generates · Reporter produces daily and weekly output artifacts.
- Scheduler → Collectors: triggers · Automated runs start collection and downstream processing.
- Scheduler → Reporter: triggers · Scheduled jobs can generate report artifacts.
- Stats → Scheduler: observes · Collection stats support operational monitoring.
- Website → Reporter: presents · The web layer exposes generated report content.