# brandon-anubis/documentation_aggregator architecture

> Architecture diagram of the GitHub repository brandon-anubis/documentation_aggregator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/brandon-anubis/documentation_aggregator
- Repository: https://github.com/brandon-anubis/documentation_aggregator

## Overview

##### Project Type and Purpose
- This is a full-stack web application (“Documentation Aggregator”)
- Frontend: React/TypeScript SPA
- Backend: Python service (likely FastAPI or Flask)
- Purpose: crawl, clean, dedupe, and aggregate documentation from external sites and present it via a web interface

##### File Structure
Top-level:
- backend/ – Python service
– Dockerfile, requirements.txt, config.py, schemas.py
– src/main.py (entry point), database.py, exceptions.py
– processors/ (content_processor.py)
– utils/ (crawler, content_cleaner, deduplication, file_manager, helpers, link_extractor, marketing_detector, sitemap_parser, validation, logger)
– web_clipper.py
- frontend/ – React app
– Dockerfile, package.json, tsconfig.json
– public/ (index.html, manifest.json, robots.txt)
– src/ (App.tsx, components, pages, config.ts, theme.ts, types.ts)
- docker-compose.yml – orchestrates frontend & backend
- main.py – possibly local orchestration or CLI entry

##### Extract Insights from README
- Single-line description: “A Documentation Aggregator”
- No explicit architecture notes, so rely on folder conventions and typical web-app patterns

##### Draft the High-Level Components
a. Client (Browser)
– React SPA served from frontend container
b. Frontend Container
– React app (App.tsx, pages, components)
– Static assets (public/)
– Communicates over HTTP/HTTPS
c. Backend Container
– Python service (main.py) exposing REST or GraphQL
– Modules: API layer, processors, utils
– web_clipper: orchestrates crawl/ingest
d. Database
– Persist crawled docs, metadata, deduplication indexes
– (config.py defines connection)
e. File Storage (optional)
– Local FS or S3 for raw documents & processed assets
f. External Services
– Third-party websites (to crawl)
– Authentication services (if any)

##### Relationships and Data Flows
- User → (1) HTTP request → Frontend
- Frontend → (2) API call → Backend
- Backend → (3) DB read/write → Database
- Backend → (4) File I/O → File Storage
- Backend → (5) HTTP fetch → External Websites
- Backend → (6) Internal modules
– utils/crawler → utils/link_extractor → utils/sitemap_parser → utils/content_cleaner → processors/content_processor → utils/deduplication
- Backend → (7) returns processed docs → Frontend → renders → User

##### Architectural Patterns
- Layered architecture: Presentation (frontend), API layer, business logic (processors+utils), data access (database, file_manager)
- Modular backend: each utility in its own module for single responsibility
- Containerized micro-services: two containers defined in docker-compose

##### Relevant Technologies
- Frontend: React, TypeScript, CSS (App.css, theme.ts), Vite or CRA (inferred)
- Backend: Python3, dependencies in requirements.txt (e.g., FastAPI/Flask, SQLAlchemy), Docker
- Orchestration: docker-compose
- Database: (Postgres/MySQL/SQLite as per config.py)
- Storage: local or cloud (S3, via file_manager)

##### Deliverables for the Principal Software Engineer
- A short annotation or callout per major flow (e.g., “User triggers Upload → frontend calls POST /documents → backend invokes web_clipper → data saved in DB + file store”).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User and Client
    User((User)):::external
    subgraph "Frontend Container"
        direction TB
        Browser["Client (React SPA)"]:::frontend
        subgraph "Static Assets"
            StaticHTML["index.html"]:::frontend
            StaticRobots["robots.txt"]:::frontend
        end
        subgraph "Components & Pages"
            Layout["Layout.tsx"]:::frontend
            Sidebar["Sidebar.tsx"]:::frontend
            subgraph "Pages"
                Home["Home.tsx"]:::frontend
                Organizations["Organizations.tsx"]:::frontend
                Results["Results.tsx"]:::frontend
                Upload["Upload.tsx"]:::frontend
                Settings["Settings.tsx"]:::frontend
            end
        end
        subgraph "Config & Types"
            ConfigFile["config.ts"]:::frontend
            Theme["theme.ts"]:::frontend
            Types["types.ts"]:::frontend
        end
        subgraph "Build & Dependencies"
            Package["package.json"]:::frontend
            TSConfig["tsconfig.json"]:::frontend
            FrontendDocker["Dockerfile"]:::frontend
        end
    end

    %% Backend Container
    subgraph "Backend Container"
        direction TB
        Entry["main.py"]:::business
        ConfigB["config.py"]:::business
        Schemas["schemas.py"]:::business
        Exceptions["exceptions.py"]:::business
        DatabaseInt["database.py"]:::dataaccess
        WebClipper["web_clipper.py"]:::business
        Processor["content_processor.py"]:::business
        subgraph "Utilities"
            Crawler["crawler.py"]:::business
            Sitemap["sitemap_parser.py"]:::business
            LinkExtractor["link_extractor.py"]:::business
            ContentCleaner["content_cleaner.py"]:::business
            Validation["validation.py"]:::business
            Deduplication["deduplication.py"]:::business
            FileManager["file_manager.py"]:::business
            Helpers["helpers.py"]:::business
            Logger["logger.py"]:::business
            MarketingDetector["marketing_detector.py"]:::business
            InputHandler["input_handler.py"]:::business
        end
        Requirements["requirements.txt"]:::infra
        BackendDocker["Dockerfile"]:::infra
    end

    %% Data & External
    DBStore(("Database")):::dataaccess
    FileStorage(("File Storage")):::dataaccess
    ExternalWebsites(("External Websites")):::external

    %% Orchestration
    subgraph "Orchestration"
        Compose["docker-compose.yml"]:::infra
        MainCLI["main.py"]:::infra
    end

    %% Connections
    User -->|"opens UI"| Browser
    Browser -->|"loads"| StaticHTML
    Browser -->|"loads"| StaticRobots
    Browser -->|"renders"| Layout
    Browser -->|"renders"| Sidebar
    Browser -->|"navigates"| Home
    Browser -->|"navigates"| Organizations
    Browser -->|"navigates"| Results
    Browser -->|"navigates"| Upload
    Browser -->|"navigates"| Settings
    Browser -->|"uses config"| ConfigFile
    Browser -->|"uses theme/types"| Theme
    Browser -->|"uses theme/types"| Types
    Browser -->|"bundle"| Package
    Browser -->|"bundle"| TSConfig
    Browser -->|"bundle"| FrontendDocker

    Browser -->|"HTTP requests"| Entry
    Entry -->|imports| ConfigB
    Entry -->|imports| Schemas
    Entry -->|imports| Exceptions
    Entry -->|"DB operations"| DatabaseInt
    Entry -->|"orchestrates"| WebClipper
    WebClipper -->|"calls"| Crawler
    Crawler -->|"parses sitemap"| Sitemap
    Sitemap -->|"extracts links"| LinkExtractor
    LinkExtractor -->|"cleans"| ContentCleaner
    ContentCleaner -->|"validates"| Validation
    ContentCleaner -->|"processes"| Processor
    Processor -->|"deduplicates"| Deduplication
    Deduplication -->|"stores files"| FileManager
    FileManager -->|"logs/helpers"| Helpers
    Helpers -->|"logs"| Logger
    Processor -->|"flags marketing"| MarketingDetector
    FileManager -->|"handles inputs"| InputHandler
    Entry -->|"reads/writes"| DBStore
    Entry -->|"file I/O"| FileStorage
    Entry -->|"fetches"| ExternalWebsites

    Compose -->|starts| Browser
    Compose -->|starts| Entry
    MainCLI -->|runs tasks| Entry

    %% Click Events
    click Browser "https://github.com/brandon-anubis/documentation_aggregator/blob/main/frontend/src/App.tsx"
    click Browser "https://github.com/brandon-anubis/documentation_aggregator/blob/main/frontend/src/index.tsx"
    click StaticHTML "https://github.com/brandon-anubis/documentation_aggregator/blob/main/frontend/public/index.html"
    click StaticRobots "https://github.com/brandon-anubis/documentation_aggregator/blob/main/frontend/public/robots.txt"
    click Layout "https://github.com/brandon-anubis/documentation_aggregator/blob/main/frontend/src/components/Layout.tsx"
    click Sidebar "https://github.com/brandon-anubis/documentation_aggregator/blob/main/frontend/src/components/Sidebar.tsx"
    click Home "https://github.com/brandon-anubis/documentation_aggregator/blob/main/frontend/src/pages/Home.tsx"
    click Organizations "https://github.com/brandon-anubis/documentation_aggregator/blob/main/frontend/src/pages/Organizations.tsx"
    click Results "https://github.com/brandon-anubis/documentation_aggregator/blob/main/frontend/src/pages/Results.tsx"
    click Upload "https://github.com/brandon-anubis/documentation_aggregator/blob/main/frontend/src/pages/Upload.tsx"
    click Settings "https://github.com/brandon-anubis/documentation_aggregator/blob/main/frontend/src/pages/Settings.tsx"
    click ConfigFile "https://github.com/brandon-anubis/documentation_aggregator/blob/main/frontend/src/config.ts"
    click Theme "https://github.com/brandon-anubis/documentation_aggregator/blob/main/frontend/src/theme.ts"
    click Types "https://github.com/brandon-anubis/documentation_aggregator/blob/main/frontend/src/types.ts"
    click Package "https://github.com/brandon-anubis/documentation_aggregator/blob/main/frontend/package.json"
    click TSConfig "https://github.com/brandon-anubis/documentation_aggregator/blob/main/frontend/tsconfig.json"
    click FrontendDocker "https://github.com/brandon-anubis/documentation_aggregator/tree/main/frontend/Dockerfile"
    click Entry "https://github.com/brandon-anubis/documentation_aggregator/blob/main/backend/src/main.py"
    click ConfigB "https://github.com/brandon-anubis/documentation_aggregator/blob/main/backend/config.py"
    click Schemas "https://github.com/brandon-anubis/documentation_aggregator/blob/main/backend/schemas.py"
    click Exceptions "https://github.com/brandon-anubis/documentation_aggregator/blob/main/backend/src/exceptions.py"
    click DatabaseInt "https://github.com/brandon-anubis/documentation_aggregator/blob/main/backend/src/database.py"
    click WebClipper "https://github.com/brandon-anubis/documentation_aggregator/blob/main/backend/src/web_clipper.py"
    click Processor "https://github.com/brandon-anubis/documentation_aggregator/blob/main/backend/src/processors/content_processor.py"
    click Crawler "https://github.com/brandon-anubis/documentation_aggregator/blob/main/backend/src/utils/crawler.py"
    click Sitemap "https://github.com/brandon-anubis/documentation_aggregator/blob/main/backend/src/utils/sitemap_parser.py"
    click LinkExtractor "https://github.com/brandon-anubis/documentation_aggregator/blob/main/backend/src/utils/link_extractor.py"
    click ContentCleaner "https://github.com/brandon-anubis/documentation_aggregator/blob/main/backend/src/utils/content_cleaner.py"
    click Validation "https://github.com/brandon-anubis/documentation_aggregator/blob/main/backend/src/utils/validation.py"
    click Deduplication "https://github.com/brandon-anubis/documentation_aggregator/blob/main/backend/src/utils/deduplication.py"
    click FileManager "https://github.com/brandon-anubis/documentation_aggregator/blob/main/backend/src/utils/file_manager.py"
    click Helpers "https://github.com/brandon-anubis/documentation_aggregator/blob/main/backend/src/utils/helpers.py"
    click Logger "https://github.com/brandon-anubis/documentation_aggregator/blob/main/backend/src/utils/logger.py"
    click MarketingDetector "https://github.com/brandon-anubis/documentation_aggregator/blob/main/backend/src/utils/marketing_detector.py"
    click InputHandler "https://github.com/brandon-anubis/documentation_aggregator/blob/main/backend/src/utils/input_handler.py"
    click Requirements "https://github.com/brandon-anubis/documentation_aggregator/blob/main/backend/requirements.txt"
    click BackendDocker "https://github.com/brandon-anubis/documentation_aggregator/tree/main/backend/Dockerfile"
    click Compose "https://github.com/brandon-anubis/documentation_aggregator/blob/main/docker-compose.yml"
    click MainCLI "https://github.com/brandon-anubis/documentation_aggregator/blob/main/main.py"

    %% Styles
    classDef frontend fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef business fill:#90EE90,stroke:#333,stroke-width:1px
    classDef dataaccess fill:#FFA500,stroke:#333,stroke-width:1px
    classDef external fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef infra fill:#E6E6FA,stroke:#333,stroke-width:1px
```
