# dhruval30/doc-analyse architecture

> Architecture diagram of the GitHub repository dhruval30/doc-analyse, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dhruval30/doc-analyse
- Repository: https://github.com/dhruval30/doc-analyse

## Overview

##### Project type and purpose
- Type: A command-line / backend utility (no GUI) for improving MoEngage documentation.
- Purpose:
– Scrape dynamic web articles on help.moengage.com.
– Analyze the cleaned text through LLM-powered evaluation modules (readability, structure, completeness, style).
– Emit structured JSON improvement suggestions.

##### File structure
- Top-level:
– Jupyter notebooks (analysis.ipynb, basic-logic.ipynb, scraper.ipynb) for prototyping.
– JSON/CSV artifacts (extracted_articles…, final.csv).
– A `scraper/` folder containing:
- app.py (entry point for scraper)
- utils.py (helper functions)
- requirements.txt (Playwright + libraries)
- Notice that the “analyzer” logic lives partly in notebooks and will eventually be codified in a utility module.

##### README for additional insights
- Two-step pipeline:
1. Scraper: Playwright browser automation → HTML cleanup → plaintext extraction
2. Analyzer: LLM API calls (OpenAI/Claude) with structured prompts → JSON suggestions
- External services:
– help.moengage.com (source)
– LLM provider APIs
- Environment:
– Playwright browsers
– LLM API keys (env vars)

##### Main components & their interactions

[User / Orchestrator]
↓ “run scraper”
[Scraper Module]
- app.py
- utils.py
- Playwright Browser Driver
↓ “cleaned text”
[Data Store / Temporary Files]
↓ “pass text”
[Analyzer Module]
- Notebook or utility
- Prompt templates for Readability, Structure, Completeness, Style
- LLM API Client (OpenAI / Claude)
↓ “JSON report”
[Output]
- JSON suggestions file
- Optional CSV or notebook visualization

Also show:
- External System: help.moengage.com
←browser requests/by Playwright→ [Scraper Module]
- External System: LLM API endpoints
←HTTP calls→ [Analyzer Module]

##### Architectural patterns & design principles
- Modular pipeline: clear separation of concerns (scraping vs. analysis).
- Extensibility points:
– Add new evaluation criteria in analyzer.
– Swap in a different scraping strategy (e.g., Selenium).
- Stateless components: each run is independent, uses flat files to pass data.

##### Technologies, frameworks, libraries
– Python 3.x
– Playwright (browser automation)
– Jupyter notebooks (prototyping)
– LLM API clients (OpenAI, Claude)
– JSON/CSV for I/O
– Environment variables for credentials

## Diagram (Mermaid)

```mermaid
flowchart TD
    User["User / Orchestrator"]:::orchestration

    ExternalMoEngage[(help.moengage.com)]:::external
    ExternalLLM[(LLM API Endpoints)]:::external

    subgraph "Scraper Module"
        direction TB
        App["app.py"]:::module
        Utils["utils.py"]:::module
        Req["requirements.txt"]:::module
        BrowserDriver["Playwright Browser Driver"]:::module
    end

    subgraph "Data Store / Temporary Files"
        direction TB
        JSONFile["extracted_articles_complete.json"]:::datastore
        CSVTemp["extracted_articles_complete_summary.csv"]:::datastore
    end

    subgraph "Analyzer Module"
        direction TB
        PromptTemplates["Prompt Templates (Readability, Structure, Completeness, Style)"]:::module
        LLMClient["LLM API Client (OpenAI / Claude)"]:::module
    end

    subgraph "Output"
        direction TB
        JSONReport["JSON suggestions file"]:::datastore
        FinalCSV["final.csv"]:::datastore
    end

    subgraph "Prototyping Notebooks"
        direction TB
        ScraperNB["scraper.ipynb"]:::prototype
        AnalysisNB["analysis.ipynb"]:::prototype
        BasicNB["basic-logic.ipynb"]:::prototype
    end

    subgraph "Supporting Artifacts"
        direction TB
        Screenshots["screenshots/"]:::support
    end

    ProjectDoc["README.md"]:::orchestration

    User -->|"1. run scraper"| App
    App -->|"HTTP GET via Playwright"| ExternalMoEngage
    ExternalMoEngage -->|"HTML response"| BrowserDriver
    BrowserDriver -->|"cleaned text payload"| JSONFile
    JSONFile -->|"pass text"| PromptTemplates
    PromptTemplates -->|"LLM JSON request/response"| ExternalLLM
    ExternalLLM -->|"LLM response"| LLMClient
    LLMClient -->|"JSON report"| JSONReport
    JSONReport --> FinalCSV

    click App "https://github.com/dhruval30/doc-analyse/blob/main/scraper/app.py"
    click Utils "https://github.com/dhruval30/doc-analyse/blob/main/scraper/utils.py"
    click Req "https://github.com/dhruval30/doc-analyse/blob/main/scraper/requirements.txt"
    click ScraperNB "https://github.com/dhruval30/doc-analyse/blob/main/scraper.ipynb"
    click AnalysisNB "https://github.com/dhruval30/doc-analyse/blob/main/analysis.ipynb"
    click BasicNB "https://github.com/dhruval30/doc-analyse/blob/main/basic-logic.ipynb"
    click JSONFile "https://github.com/dhruval30/doc-analyse/blob/main/extracted_articles_complete.json"
    click CSVTemp "https://github.com/dhruval30/doc-analyse/blob/main/extracted_articles_complete_summary.csv"
    click FinalCSV "https://github.com/dhruval30/doc-analyse/blob/main/final.csv"
    click ProjectDoc "https://github.com/dhruval30/doc-analyse/blob/main/README.md"
    click Screenshots "https://github.com/dhruval30/doc-analyse/tree/main/screenshots/"

    classDef orchestration fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef module fill:#90EE90,stroke:#333,stroke-width:1px
    classDef datastore fill:#FFFF99,stroke:#333,stroke-width:1px
    classDef external fill:#FFA500,stroke:#333,stroke-width:1px
    classDef support fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef prototype fill:#90EE90,stroke:#333,stroke-width:1px
```
