# madhavendrapandey/friction-intel architecture

> Architecture diagram of the GitHub repository madhavendrapandey/friction-intel, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/madhavendrapandey/friction-intel
- Repository: https://github.com/madhavendrapandey/friction-intel

## Overview

1. **What this repo is**
   Friction-Intel is a Python pipeline for collecting discussion data, extracting structured evidence, normalizing and grouping it, then producing human-readable reports and a small web UI over the results.

2. **Ingestion boundary**
   The first subsystem is `modules/friction/collectors/`, which adapts multiple source types into a common intake flow: GitHub Discussions, Hacker News, Reddit, RSS feeds, and local seed data. The `data/seed/*.json` files are an offline/bootstrap source for development or repeatable runs.

3. **Shared domain layer**
   `modules/friction/domain.py` appears to define the core entities used across the pipeline, while `modules/friction/paths.py` centralizes filesystem locations. These act as the shared contract between collectors, services, repositories, and reporting.

4. **Extraction and normalization pipeline**
   `modules/friction/extractors/` turns raw collected content into structured evidence using rule-based and pattern-based logic. That output is then refined by services such as normalization, vocabulary handling, and quality filtering so downstream analysis works on consistent concepts rather than source-specific text.

5. **Repository/data-access boundary**
   `modules/friction/repositories/` is the persistence layer over a SQLite database in `data/friction.db`. The repository set is fairly broad: posts, sources, snapshots, evidence, normalized evidence, candidate issues, evidence groups, contradictions, relationships, profiles, validation, traceability, and pipeline runs. This suggests the app persists both raw intake and multiple derived views of the same underlying information.

6. **Analytical services**
   The main business logic lives in `modules/friction/services/`. Key stages include collection orchestration, extraction, normalization, grouping, candidates generation, contradiction detection, relationships, profile construction, evolution tracking, and validation. These services likely form the core pipeline that transforms raw source material into analysis-ready outputs.

7. **Traceability and provenance**
   The presence of `traceability`, `validation`, `snapshots`, and `pipeline_runs` repositories indicates the system is designed to explain where derived facts came from and how they changed over time. This is important for any architecture diagram: provenance is not incidental, it is a first-class part of the model.

8. **Reporting outputs**
   `modules/friction/services/reporting.py` writes reports into `modules/friction/reports/` as both `.json` and `.md` artifacts, organized by daily, weekly, and monthly time windows. These generated files are part of the product surface, not just internal logs.

9. **Web presentation layer**
   The `web/` directory is a read-only UI layer backed by Jinja templates. The templates focus on landing, reports listing/detail, friction overview, evidence source views, profile detail, and profile evidence. That implies the web app is mainly a presentation and navigation layer over precomputed pipeline data rather than a primary write path.

10. **Orchestration and runtime shape**
   `modules/friction/runner.py` is the likely top-level coordinator that wires collectors, services, repositories, and reporting together. `requirements.txt` indicates a Python runtime with SQLite-backed persistence, and the tests cover the major analytical/reporting behaviors, which reinforces that the repo is organized around a batch-style data pipeline plus a lightweight UI.

Main entry points: [`modules/friction/runner.py`](https://github.com/madhavendrapandey/friction-intel/blob/main/modules/friction/runner.py) (Runner), [`data/friction.db`](https://github.com/madhavendrapandey/friction-intel/blob/main/data/friction.db) (SQLite DB), [`modules/friction/services/normalization.py`](https://github.com/madhavendrapandey/friction-intel/blob/main/modules/friction/services/normalization.py) (Normalization), [`web/main.py`](https://github.com/madhavendrapandey/friction-intel/blob/main/web/main.py) (Web UI)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["Ingestion"]
  node_collection_sources["Sources<br/>collectors"]
  node_seed_data["Seed data<br/>offline inputs"]
  node_collectors_base["Collector base<br/>adapter base<br/>[base.py]"]
end

subgraph group_core["Core pipeline"]
  node_runner["Runner<br/>orchestrator<br/>[runner.py]"]
  node_domain["Domain<br/>entities<br/>[domain.py]"]
  node_paths["Paths<br/>filesystem map<br/>[paths.py]"]
  node_collection_service["Collecting<br/>[collection.py]"]
  node_extraction_service["Extraction<br/>[extraction.py]"]
  node_normalization_service["Normalization<br/>[normalization.py]"]
  node_grouping_service["Grouping<br/>[grouping.py]"]
  node_analysis_service{{"Analysis<br/>service bundle"}}
  node_quality_filter["Quality filter<br/>[quality_filter.py]"]
  node_vocabulary_service["Vocabulary<br/>[vocabulary.py]"]
  node_traceability_service["Traceability<br/>[traceability.py]"]
end

subgraph group_data["Persistence"]
  node_evidence_repo[("Evidence")]
  node_normalized_repo[("Normalized")]
  node_provenance_repo[("Provenance")]
  node_analysis_repos[("Derived state<br/>repository set")]
  node_db[("SQLite DB<br/>storage<br/>[friction.db]")]
end

subgraph group_output["Outputs"]
  node_reporting_service["Reporting<br/>[reporting.py]"]
  node_reports_artifacts["Reports<br/>generated artifacts"]
  node_web_app["Web UI<br/>presentation<br/>[main.py]"]
  node_web_templates["Templates<br/>views"]
end

node_runner -->|"orchestrates"| node_collection_service
node_runner -->|"runs"| node_extraction_service
node_runner -->|"runs"| node_normalization_service
node_runner -->|"runs"| node_grouping_service
node_runner -->|"drives"| node_analysis_service
node_runner -->|"publishes"| node_reporting_service
node_collection_service -->|"ingests"| node_collection_sources
node_collection_sources -->|"implements"| node_collectors_base
node_collection_sources -.->|"includes"| node_seed_data
node_collection_service -->|"uses"| node_domain
node_extraction_service -->|"writes"| node_evidence_repo
node_extraction_service -->|"maps to"| node_domain
node_normalization_service -->|"writes"| node_normalized_repo
node_normalization_service -->|"uses"| node_vocabulary_service
node_normalization_service -->|"applies"| node_quality_filter
node_grouping_service -->|"updates"| node_analysis_repos
node_analysis_service -->|"reads/writes"| node_analysis_repos
node_analysis_service -->|"records"| node_traceability_service
node_analysis_service -->|"stores"| node_provenance_repo
node_traceability_service -->|"persists"| node_provenance_repo
node_reporting_service -->|"generates"| node_reports_artifacts
node_reporting_service -->|"uses"| node_paths
node_web_app -->|"renders"| node_web_templates
node_web_app -->|"reads"| node_reports_artifacts
node_web_app -->|"reads"| node_analysis_repos
node_db -->|"backing store"| node_evidence_repo
node_db -->|"backing store"| node_normalized_repo
node_db -->|"backing store"| node_analysis_repos
node_db -->|"backing store"| node_provenance_repo
node_paths -->|"locates"| node_reports_artifacts

click node_runner "https://github.com/madhavendrapandey/friction-intel/blob/main/modules/friction/runner.py"
click node_domain "https://github.com/madhavendrapandey/friction-intel/blob/main/modules/friction/domain.py"
click node_paths "https://github.com/madhavendrapandey/friction-intel/blob/main/modules/friction/paths.py"
click node_collection_service "https://github.com/madhavendrapandey/friction-intel/blob/main/modules/friction/services/collection.py"
click node_extraction_service "https://github.com/madhavendrapandey/friction-intel/blob/main/modules/friction/services/extraction.py"
click node_normalization_service "https://github.com/madhavendrapandey/friction-intel/blob/main/modules/friction/services/normalization.py"
click node_grouping_service "https://github.com/madhavendrapandey/friction-intel/blob/main/modules/friction/services/grouping.py"
click node_quality_filter "https://github.com/madhavendrapandey/friction-intel/blob/main/modules/friction/services/quality_filter.py"
click node_vocabulary_service "https://github.com/madhavendrapandey/friction-intel/blob/main/modules/friction/services/vocabulary.py"
click node_traceability_service "https://github.com/madhavendrapandey/friction-intel/blob/main/modules/friction/services/traceability.py"
click node_collection_sources "https://github.com/madhavendrapandey/friction-intel/tree/main/modules/friction/collectors"
click node_seed_data "https://github.com/madhavendrapandey/friction-intel/tree/main/data/seed"
click node_collectors_base "https://github.com/madhavendrapandey/friction-intel/blob/main/modules/friction/collectors/base.py"
click node_evidence_repo "https://github.com/madhavendrapandey/friction-intel/blob/main/modules/friction/repositories/evidence_repository.py"
click node_normalized_repo "https://github.com/madhavendrapandey/friction-intel/blob/main/modules/friction/repositories/normalized_evidence_repository.py"
click node_db "https://github.com/madhavendrapandey/friction-intel/blob/main/data/friction.db"
click node_reporting_service "https://github.com/madhavendrapandey/friction-intel/blob/main/modules/friction/services/reporting.py"
click node_reports_artifacts "https://github.com/madhavendrapandey/friction-intel/tree/main/modules/friction/reports"
click node_web_app "https://github.com/madhavendrapandey/friction-intel/blob/main/web/main.py"
click node_web_templates "https://github.com/madhavendrapandey/friction-intel/tree/main/web/templates"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_collection_sources,node_seed_data,node_collectors_base toneBlue
class node_runner,node_domain,node_paths,node_collection_service,node_extraction_service,node_normalization_service,node_grouping_service,node_analysis_service,node_quality_filter,node_vocabulary_service,node_traceability_service toneAmber
class node_evidence_repo,node_normalized_repo,node_provenance_repo,node_analysis_repos,node_db toneMint
class node_reporting_service,node_reports_artifacts,node_web_app,node_web_templates toneRose
```

## Components

### Ingestion

Collectors adapt external discussion sources and local seed data into the intake flow.

- [Sources](https://github.com/madhavendrapandey/friction-intel/tree/main/modules/friction/collectors): collectors · GitHub Discussions, Hacker News, Reddit, RSS, and seed intake. · `modules/friction/collectors`
- [Seed data](https://github.com/madhavendrapandey/friction-intel/tree/main/data/seed): offline inputs · Bootstrap JSON data for repeatable or offline runs. · `data/seed`
- [Collector base](https://github.com/madhavendrapandey/friction-intel/blob/main/modules/friction/collectors/base.py): adapter base · Common collector interface. · `modules/friction/collectors/base.py`

### Core pipeline

Domain model, extraction, normalization, grouping, and analysis services.

- [Runner](https://github.com/madhavendrapandey/friction-intel/blob/main/modules/friction/runner.py): orchestrator · Top-level pipeline coordinator. · `modules/friction/runner.py`
- [Domain](https://github.com/madhavendrapandey/friction-intel/blob/main/modules/friction/domain.py): entities · Shared core data model. · `modules/friction/domain.py`
- [Paths](https://github.com/madhavendrapandey/friction-intel/blob/main/modules/friction/paths.py): filesystem map · Centralized repository and artifact locations. · `modules/friction/paths.py`
- [Collecting](https://github.com/madhavendrapandey/friction-intel/blob/main/modules/friction/services/collection.py): service · Coordinates source intake across collectors. · `modules/friction/services/collection.py`
- [Extraction](https://github.com/madhavendrapandey/friction-intel/blob/main/modules/friction/services/extraction.py): service · Turns raw content into structured evidence. · `modules/friction/services/extraction.py`
- [Normalization](https://github.com/madhavendrapandey/friction-intel/blob/main/modules/friction/services/normalization.py): service · Refines evidence into consistent concepts. · `modules/friction/services/normalization.py`
- [Grouping](https://github.com/madhavendrapandey/friction-intel/blob/main/modules/friction/services/grouping.py): service · Clusters related evidence and concepts. · `modules/friction/services/grouping.py`
- **Analysis**: service bundle · Candidates, contradictions, relationships, profiles, evolution, validation.
- [Quality filter](https://github.com/madhavendrapandey/friction-intel/blob/main/modules/friction/services/quality_filter.py): service · Suppresses low-signal or unreliable evidence. · `modules/friction/services/quality_filter.py`
- [Vocabulary](https://github.com/madhavendrapandey/friction-intel/blob/main/modules/friction/services/vocabulary.py): service · Handles canonical terms and concept alignment. · `modules/friction/services/vocabulary.py`
- [Traceability](https://github.com/madhavendrapandey/friction-intel/blob/main/modules/friction/services/traceability.py): service · Tracks provenance for derived facts and transformations. · `modules/friction/services/traceability.py`

### Persistence

SQLite-backed repositories for raw, derived, and provenance data.

- [Evidence](https://github.com/madhavendrapandey/friction-intel/blob/main/modules/friction/repositories/evidence_repository.py): repository · Stores extracted evidence records. · `modules/friction/repositories/evidence_repository.py`
- [Normalized](https://github.com/madhavendrapandey/friction-intel/blob/main/modules/friction/repositories/normalized_evidence_repository.py): repository · Stores normalized evidence records. · `modules/friction/repositories/normalized_evidence_repository.py`
- **Provenance**: repository · Snapshots, traceability, validation, and pipeline run history.
- **Derived state**: repository set · Candidates, groups, contradictions, relationships, profiles, posts, sources.
- [SQLite DB](https://github.com/madhavendrapandey/friction-intel/blob/main/data/friction.db): storage · Persistent backing store for the pipeline. · `data/friction.db`

### Outputs

Generated reports and the read-only web presentation layer.

- [Reporting](https://github.com/madhavendrapandey/friction-intel/blob/main/modules/friction/services/reporting.py): service · Writes daily, weekly, and monthly JSON/Markdown reports. · `modules/friction/services/reporting.py`
- [Reports](https://github.com/madhavendrapandey/friction-intel/tree/main/modules/friction/reports): generated artifacts · Published report files under daily, weekly, and monthly windows. · `modules/friction/reports`
- [Web UI](https://github.com/madhavendrapandey/friction-intel/blob/main/web/main.py): presentation · Read-only Jinja-based interface over precomputed results. · `web/main.py`
- [Templates](https://github.com/madhavendrapandey/friction-intel/tree/main/web/templates): views · Landing, reports, evidence source, and profile pages. · `web/templates`

## Connections

- Runner → Collecting: orchestrates · Coordinates ingestion as the pipeline entrypoint.
- Runner → Extraction: runs · Executes the structured evidence pipeline.
- Runner → Normalization: runs · Applies normalization before downstream analysis.
- Runner → Grouping: runs · Clusters evidence into analysis units.
- Runner → Analysis: drives · Triggers later-stage analytical services.
- Runner → Reporting: publishes · Emits human-readable outputs after analysis.
- Collecting → Sources: ingests · Pulls from external sources and local seeds.
- Sources → Collector base: implements · Source-specific adapters share a common intake contract.
- Sources → Seed data: includes · Offline/bootstrap inputs for repeatable runs.
- Collecting → Domain: uses · Produces domain entities for downstream stages.
- Extraction → Evidence: writes · Persists extracted evidence.
- Extraction → Domain: maps to · Converts raw inputs into shared entities.
- Normalization → Normalized: writes · Stores canonicalized evidence.
- Normalization → Vocabulary: uses · Aligns terms to canonical vocabulary.
- Normalization → Quality filter: applies · Filters low-quality or noisy items.
- Grouping → Derived state: updates · Persists grouped and derived analysis state.
- Analysis → Derived state: reads/writes · Candidate, contradiction, relationship, profile, and source state.
- Analysis → Traceability: records · Preserves provenance and transformation history.
- Analysis → Provenance: stores · Captures snapshots, validation, and run metadata.
- Traceability → Provenance: persists · Writes trace and provenance records.
- Reporting → Reports: generates · Produces JSON and Markdown report artifacts.
- Reporting → Paths: uses · Resolves report output locations.
- Web UI → Templates: renders · Serves read-only pages from templates.
- Web UI → Reports: reads · Displays precomputed report outputs.
- Web UI → Derived state: reads · Shows evidence, profiles, and related derived data.
- SQLite DB → Evidence: backing store · SQLite persistence for repository data.
- SQLite DB → Normalized: backing store · SQLite persistence for repository data.
- SQLite DB → Derived state: backing store · SQLite persistence for repository data.
- SQLite DB → Provenance: backing store · SQLite persistence for repository data.
- Paths → Reports: locates · Defines report artifact directories.