How dhruval30/doc-analyse works
Project type and purpose
• Type: A command-line / backend utility (no GUI) for improving MoEngage documentation.
• Purpose:
– Scrape dynamic web articles on help.moengage.com.
– Analyze the cleaned text through LLM-powered evaluation modules (readability, structure, completeness, style).
– Emit structured JSON improvement suggestions.
File structure
• Top-level:
– Jupyter notebooks (analysis.ipynb, basic-logic.ipynb, scraper.ipynb) for prototyping.
– JSON/CSV artifacts (extracted_articles…, final.csv).
– A scraper/ folder containing:
• app.py (entry point for scraper)
• utils.py (helper functions)
• requirements.txt (Playwright + libraries)
• Notice that the “analyzer” logic lives partly in notebooks and will eventually be codified in a utility module.
README for additional insights
• Two-step pipeline:
1. Scraper: Playwright browser automation → HTML cleanup → plaintext extraction
2. Analyzer: LLM API calls (OpenAI/Claude) with structured prompts → JSON suggestions
• External services:
– help.moengage.com (source)
– LLM provider APIs
• Environment:
– Playwright browsers
– LLM API keys (env vars)
Main components & their interactions
[User / Orchestrator]
↓ “run scraper”
[Scraper Module]
• app.py
• utils.py
• Playwright Browser Driver
↓ “cleaned text”
[Data Store / Temporary Files]
↓ “pass text”
[Analyzer Module]
• Notebook or utility
• Prompt templates for Readability, Structure, Completeness, Style
• LLM API Client (OpenAI / Claude)
↓ “JSON report”
[Output]
• JSON suggestions file
• Optional CSV or notebook visualization
Also show:
• External System: help.moengage.com
←browser requests/by Playwright→ [Scraper Module]
• External System: LLM API endpoints
←HTTP calls→ [Analyzer Module]
Architectural patterns & design principles
• Modular pipeline: clear separation of concerns (scraping vs. analysis).
• Extensibility points:
– Add new evaluation criteria in analyzer.
– Swap in a different scraping strategy (e.g., Selenium).
• Stateless components: each run is independent, uses flat files to pass data.
Technologies, frameworks, libraries
– Python 3.x
– Playwright (browser automation)
– Jupyter notebooks (prototyping)
– LLM API clients (OpenAI, Claude)
– JSON/CSV for I/O
– Environment variables for credentials