How madhavendrapandey/trend_intelligence_system works
1. Repository shape
- This is a batch-oriented intelligence platform, not a single app. The core flow is: external sources are collected, filtered and analyzed, then persisted and rendered into reports and a web UI.
- The codebase is split into two main intelligence domains: trend intelligence and friction intelligence.
2. External data ingestion
- The core/collectors layer pulls raw content from multiple sources: RSS, GitHub, arXiv, Hacker News, and Reddit.
- Source-specific configuration lives in core/sources, which acts as the source registry / selectors for what gets collected.
3. Pre-processing and signal hygiene
- Before analysis, items pass through filters in core/filters:
- duplicate detection
- content quality checks
- source quality checks
- interest scoring / relevance filtering
- Reddit-specific quality filtering
- This is the first major boundary: raw internet noise is reduced to a smaller candidate set for intelligence work.
4. LLM abstraction
- core/llm/providers.py centralizes model-provider access and supports both local and API-based LLM backends.
- The README indicates support for Ollama/Qwen locally, plus cloud providers like Gemini, OpenAI, Anthropic, Groq, OpenRouter, and Together AI.
- This layer is used by downstream extraction/grouping services rather than being coupled directly to the UI or collectors.
5. Domain models
- core/models defines the primary business objects: evidence, evidence groups, friction candidates, friction profiles, validation events, and related Qwen/model abstractions.
- These models make the analysis pipeline explicit: raw items become evidence, evidence is grouped, grouped evidence informs candidates/profiles, and validation tracks confidence or review state.
6. Trend intelligence pipeline
- modules/trend is the trend-analysis subsystem.
- Its engines handle:
- signal extraction
- topic normalization
- trend scoring
- opportunity detection
- recommendation generation
- modules/trend/services/trend_intelligence_service.py likely orchestrates these engines into a coherent trend report pipeline.
7. Friction intelligence pipeline
- modules/friction is a separate analysis stack focused on friction, contradictions, and validation.
- It includes:
- evidence extraction
- evidence grouping
- contradiction analysis
- candidate generation
- relationship/evolution/profile services
- traceability and validation services
- This subsystem appears designed to turn collected observations into structured “friction intelligence” objects rather than generic trend summaries.
8. Persistence and schema evolution
- database/ contains the SQLite-backed persistence layer: repositories for posts, sources, runs, evidence, groups, candidates, profiles, validations, trends, operators, and failure tracking.
- Migrations show the system evolved from a simpler article store to a richer schema with evidence, friction, operator tooling, and trend intelligence.
- The presence of SQLITE_CUTOVER_REPORT.md and migration scripts suggests a deliberate transition from JSON storage to SQLite as the authoritative store.
9. Storage boundary and repositories
- core/storage contains storage interfaces plus JSON and SQLite implementations, so the app can work across persistence backends.
- database/repositories is the main data-access layer used by higher-level services and the website.
- This cleanly separates analysis logic from storage concerns.
10. Reporting outputs
- Reports are generated into reports/, including daily markdown/JSON artifacts and weekly briefs.
- Top-level scripts like reporter.py, analyzer.py, and main.py indicate a staged pipeline where collection, analysis, and report generation can run independently or end-to-end.
- There are also HTML outputs such as evidence_output.html and operator_output.html, suggesting artifact-based inspection is part of the workflow.
11. Web application and operator console
- website/ is a presentation layer over the stored intelligence.
- It uses routes, templates, static assets, and service abstractions to render pages for evidence, sources, profiles, reports, and operator workflows.
- The operator section includes dashboard, explorer, pipeline monitor, SQL studio, schema browser, and table browser views, which makes the persistence layer inspectable for debugging and quality control.
12. Operational tooling and metrics
- scheduler.py suggests automated periodic runs.
- stats/ tracks collection statistics, which likely feed monitoring or quality reporting.
- database/compare_report_parity.py and migration utilities indicate the system includes tooling for validating the new database-backed pipeline against prior report outputs.
13. Primary end-to-end data flow
- External sources → collectors → filters/deduplication → extraction/LLM analysis → trend/friction engines → SQLite repositories → reports and website.
- The most important architectural boundaries are: ingestion vs analysis, analysis vs persistence, and persistence vs presentation.
14. What to draw in an architecture diagram
- Show two parallel intelligence lanes sharing the same ingestion and storage foundation: Trend Intelligence and Friction Intelligence.
- Put the database/repository layer underneath both, with the website and reporting tools reading from it.
- Highlight the LLM provider boundary as a shared dependency used selectively by analysis services, not by collectors or UI code.
Main entry points: repositories Repositories · extractors Evidence Extractors · services Friction Services · filters Filters
Components 19
Ingestion
Source registry, collectors, and signal hygiene before analysis.
- Sources · Source selectors and configuration for what gets collected.
core/sources - Collectors · Fetches raw content from RSS, GitHub, arXiv, Hacker News, and Reddit.
core/collectors - Filters · Duplicate, quality, source, interest, and Reddit-specific filtering.
core/filters
Analysis
Parallel trend and friction intelligence pipelines.
- LLM Providers · Centralized access to local and hosted model backends.
core/llm/providers.py - Domain Models · Evidence, groups, candidates, profiles, and validation events.
core/models - Trend Service · Coordinates the trend intelligence pipeline.
modules/trend/services/trend_intelligence_service.py - Trend Engines · Signal extraction, topic normalization, scoring, opportunity detection, and recommendations.
modules/trend/engines - Friction Runner · Entry point for the friction intelligence workflow.
modules/friction/runner.py - Evidence Extractors · Builds evidence from collected material, including LLM-assisted extraction.
modules/friction/extractors - Friction Services · Grouping, contradiction, candidate, relationship, evolution, traceability, and validation services.
modules/friction/services
Persistence
SQLite storage, repositories, and schema evolution.
- SQLite Storage · Shared persistence interface implementation for SQLite.
core/storage/sqlite_storage.py - JSON Storage · Legacy or alternate storage implementation behind the same interface.
core/storage/json_storage.py - Repositories · Primary database repositories for evidence, trends, profiles, runs, and validation.
database/repositories - Migrations · Database schema and cutover support for evidence, friction, operator, and trend data.
database/migrations
Presentation
Reports and web UI for browsing intelligence artifacts.
Connections 22
- Sources → Collectors: selects · Source registry determines which feeds and endpoints are collected.
- Collectors → Filters: raw items · Collected content is cleaned and reduced before analysis.
- Filters → Evidence Extractors: candidate items · Filtered items feed evidence extraction.
- Filters → Trend Service: signals · Clean items become inputs to trend intelligence.
- LLM Providers → Evidence Extractors: assist · LLM backends support evidence extraction where needed.
- LLM Providers → Friction Services: assist · LLM-backed grouping and analysis services use the shared provider layer.
- Domain Models → Evidence Extractors: shape evidence · Extractors produce evidence and related domain objects.
- Domain Models → Friction Services: govern analysis · Services operate on evidence groups, candidates, profiles, and validations.
- Trend Service → Trend Engines: orchestrates · Trend service composes the trend pipeline stages.
- Evidence Extractors → Friction Services: feeds · Extracted evidence flows into grouping, contradiction, and validation services.
- Friction Services → Repositories: persist · Friction intelligence artifacts are stored through repositories.
- Trend Engines → Repositories: persist · Trend outputs are written to the database layer.
- SQLite Storage → Repositories: backing store · Repositories read and write through the SQLite storage boundary.
- JSON Storage → Repositories: legacy store · Alternative storage path used by the storage interface.
- Migrations → Repositories: evolves schema · Schema changes support the repository layer as the data model grows.
- Repositories → Reports: source data · Reports are rendered from persisted intelligence objects.
- Repositories → Website: serve data · The web app reads persisted records for browsing and operator views.
- Reporter → Reports: generates · Reporter produces daily and weekly output artifacts.
- Scheduler → Collectors: triggers · Automated runs start collection and downstream processing.
- Scheduler → Reporter: triggers · Scheduled jobs can generate report artifacts.
- Stats → Scheduler: observes · Collection stats support operational monitoring.
- Website → Reporter: presents · The web layer exposes generated report content.