madhavendrapandey/friction-intel

1. What this repo is

Friction-Intel is a Python pipeline for collecting discussion data, extracting structured evidence, normalizing and grouping it, then producing human-readable reports and a small web UI over the results.

2. Ingestion boundary

The first subsystem is modules/friction/collectors/, which adapts multiple source types into a common intake flow: GitHub Discussions, Hacker News, Reddit, RSS feeds, and local seed data. The data/seed/*.json files are an offline/bootstrap source for development or repeatable runs.

3. Shared domain layer

modules/friction/domain.py appears to define the core entities used across the pipeline, while modules/friction/paths.py centralizes filesystem locations. These act as the shared contract between collectors, services, repositories, and reporting.

4. Extraction and normalization pipeline

modules/friction/extractors/ turns raw collected content into structured evidence using rule-based and pattern-based logic. That output is then refined by services such as normalization, vocabulary handling, and quality filtering so downstream analysis works on consistent concepts rather than source-specific text.

5. Repository/data-access boundary

modules/friction/repositories/ is the persistence layer over a SQLite database in data/friction.db. The repository set is fairly broad: posts, sources, snapshots, evidence, normalized evidence, candidate issues, evidence groups, contradictions, relationships, profiles, validation, traceability, and pipeline runs. This suggests the app persists both raw intake and multiple derived views of the same underlying information.

6. Analytical services

The main business logic lives in modules/friction/services/. Key stages include collection orchestration, extraction, normalization, grouping, candidates generation, contradiction detection, relationships, profile construction, evolution tracking, and validation. These services likely form the core pipeline that transforms raw source material into analysis-ready outputs.

7. Traceability and provenance

The presence of traceability, validation, snapshots, and pipeline_runs repositories indicates the system is designed to explain where derived facts came from and how they changed over time. This is important for any architecture diagram: provenance is not incidental, it is a first-class part of the model.

8. Reporting outputs

modules/friction/services/reporting.py writes reports into modules/friction/reports/ as both .json and .md artifacts, organized by daily, weekly, and monthly time windows. These generated files are part of the product surface, not just internal logs.

9. Web presentation layer

The web/ directory is a read-only UI layer backed by Jinja templates. The templates focus on landing, reports listing/detail, friction overview, evidence source views, profile detail, and profile evidence. That implies the web app is mainly a presentation and navigation layer over precomputed pipeline data rather than a primary write path.

10. Orchestration and runtime shape

modules/friction/runner.py is the likely top-level coordinator that wires collectors, services, repositories, and reporting together. requirements.txt indicates a Python runtime with SQLite-backed persistence, and the tests cover the major analytical/reporting behaviors, which reinforces that the repo is organized around a batch-style data pipeline plus a lightweight UI.