# hhshanto/automated_moodle_scraper architecture

> Architecture diagram of the GitHub repository hhshanto/automated_moodle_scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hhshanto/automated_moodle_scraper
- Repository: https://github.com/hhshanto/automated_moodle_scraper

## Overview

1. **What this repo is**
   - This is a browser automation pipeline for Moodle, centered on a Playwright-driven browser session that can be controlled either by an external MCP client or by an internal Azure OpenAI agent loop.
   - The architecture is split between **tool definitions**, **browser/auth session management**, **LLM orchestration**, and **export/output**.

2. **Primary entry point**
   - `main.py` is the runtime switchboard.
   - It supports three modes: default **MCP server**, `--agent` for the standalone Azure OpenAI tool-calling loop, and `--test-login` for a manual browser login smoke test.
   - This means the repo is not a single workflow; it is a set of reusable capabilities exposed through different control planes.

3. **Shared automation core**
   - `src/moodle_scraper/tools.py` is the central boundary for browser actions.
   - The README states this is the “single source of truth” for tools, and both the MCP server and the standalone agent call into it.
   - That design prevents duplicated logic between “LLM client controls browser” and “internal agent controls browser” paths.

4. **Browser/session layer**
   - `src/moodle_scraper/browser.py` likely owns the Playwright browser lifecycle: launch, page context, screenshots, and shutdown.
   - This is the execution substrate for all scraping actions and the main stateful boundary in the system.
   - The current implemented tools indicate the browser layer already supports navigation, visible-page text extraction, screenshots, status checks, and closure.

5. **Authentication boundary**
   - `src/moodle_scraper/auth.py` encapsulates Moodle login, including DFN-AAI SSO flow handling.
   - This is a separate subsystem from generic browsing because it has site-specific selectors and identity-provider behavior.
   - `main.py --test-login` exists specifically to validate this boundary without any LLM involvement.

6. **LLM-driven orchestration**
   - `src/moodle_scraper/agent.py` implements the standalone agent loop: it sends the goal to Azure OpenAI, receives tool-call decisions, executes tools, and repeats until completion.
   - `src/moodle_scraper/parser.py` is the Azure OpenAI client wrapper used by that loop.
   - `src/moodle_scraper/prompt.txt` likely contains the agent’s system prompt or instructions, making prompt behavior part of the orchestration layer rather than the tool layer.

7. **MCP server integration**
   - `src/moodle_scraper/mcp_server.py` exposes the same tool surface through Model Context Protocol.
   - This turns the scraper into a service that external LLM clients such as Claude Desktop or VS Code Copilot can control.
   - Architecturally, MCP is just an alternate control interface over the same underlying automation core.

8. **Data model and parsing layer**
   - `src/moodle_scraper/models.py` is intended for structured quiz/course data, likely via Pydantic.
   - `src/moodle_scraper/parser.py` sits at the boundary where unstructured browser/page content is turned into model-driven, LLM-assisted structured output.
   - In the current repo state, this is more scaffolded than fully realized, since the README marks quiz scraping/export as planned.

9. **Export/output subsystem**
   - `src/moodle_scraper/exporter.py` is the output boundary for XML, HTML, and screenshots.
   - Screenshots are already part of the implemented workflow and are stored under `output/screenshots/`.
   - The planned XML/HTML exports indicate a future stage where scraped Moodle quiz data becomes portable artifacts, not just browser state.

10. **Utilities and cross-cutting concerns**
   - `src/moodle_scraper/utils.py` centralizes logging, timestamps, and output path handling.
   - `.env`-driven configuration controls Moodle credentials, Azure OpenAI settings, headless mode, and output location.
   - This keeps environment-specific concerns outside the browser and agent logic.

11. **External systems and runtime dependencies**
   - The browser automation depends on **Playwright + Chromium**.
   - Authentication targets a **Moodle instance with DFN-AAI SSO**.
   - The agent mode depends on **Azure OpenAI GPT-4o**, specifically for tool-calling and vision-enabled interaction.
   - The MCP mode depends on **MCP-compatible clients** rather than a direct OpenAI loop.

12. **Current maturity of the system**
   - The repository is partly operational and partly scaffolded.
   - Implemented today: login, navigation, screenshots, page text extraction, status checks, browser shutdown, MCP exposure, and Azure OpenAI orchestration.
   - Planned next: course discovery, quiz listing, quiz scraping, and export to XML/HTML.

13. **Test coverage boundary**
   - `tests/test_auth.py`, `tests/test_parser.py`, and `tests/test_azure_openai.py` validate the two most fragile seams: Moodle authentication and Azure OpenAI integration.
   - This reflects the architecture’s two biggest external dependencies: identity-provider login and model/API interaction.

Main entry points: [`src/moodle_scraper/tools.py`](https://github.com/hhshanto/automated_moodle_scraper/blob/main/src/moodle_scraper/tools.py) (Tools), [`src/moodle_scraper/parser.py`](https://github.com/hhshanto/automated_moodle_scraper/blob/main/src/moodle_scraper/parser.py) (Azure client), [`src/moodle_scraper/agent.py`](https://github.com/hhshanto/automated_moodle_scraper/blob/main/src/moodle_scraper/agent.py) (Agent loop), [`main.py`](https://github.com/hhshanto/automated_moodle_scraper/blob/main/main.py) (main.py)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Runtime & control"]
  node_main_py["main.py<br/>entrypoint<br/>[main.py]"]
  node_mcp_server["MCP server<br/>[mcp_server.py]"]
end

subgraph group_automation["Browser automation"]
  node_tools["Tools<br/>browser actions<br/>[tools.py]"]
  node_browser["Browser<br/>playwright session<br/>[browser.py]"]
  node_auth["Auth<br/>login flow<br/>[auth.py]"]
  node_utils["Utils<br/>cross-cutting<br/>[utils.py]"]
end

subgraph group_orchestration["LLM orchestration"]
  node_agent["Agent loop<br/>orchestrator<br/>[agent.py]"]
  node_parser["Azure client<br/>llm client<br/>[parser.py]"]
  node_prompt["Prompt<br/>instructions<br/>[prompt.txt]"]
end

subgraph group_data["Data & output"]
  node_models[("Models<br/>data model<br/>[models.py]")]
  node_exporter["Exporter<br/>output<br/>[exporter.py]"]
end

node_tests_auth["Auth tests<br/>[test_auth.py]"]
node_tests_parser["Parser tests<br/>[test_parser.py]"]
node_tests_azure["Azure tests"]
node_readme["README<br/>docs<br/>[README.md]"]

node_main_py -->|"default mode"| node_mcp_server
node_main_py -->|"--agent"| node_agent
node_main_py -->|"--test-login"| node_auth
node_mcp_server -->|"exposes"| node_tools
node_agent -->|"calls"| node_parser
node_agent -->|"loads"| node_prompt
node_agent -->|"executes"| node_tools
node_parser -.->|"tool schema"| node_tools
node_tools -->|"controls"| node_browser
node_tools -->|"login"| node_auth
node_tools -->|"uses"| node_utils
node_browser -->|"uses"| node_utils
node_auth -->|"drives"| node_browser
node_parser -.->|"structures"| node_models
node_exporter -->|"serializes"| node_models
node_exporter -->|"writes via"| node_utils
node_tests_auth -.->|"validates"| node_auth
node_tests_parser -.->|"validates"| node_parser
node_tests_azure -.->|"validates"| node_parser
node_readme -.->|"documents"| node_main_py
node_readme -.->|"documents"| node_tools
node_readme -.->|"documents"| node_exporter

click node_main_py "https://github.com/hhshanto/automated_moodle_scraper/blob/main/main.py"
click node_mcp_server "https://github.com/hhshanto/automated_moodle_scraper/blob/main/src/moodle_scraper/mcp_server.py"
click node_agent "https://github.com/hhshanto/automated_moodle_scraper/blob/main/src/moodle_scraper/agent.py"
click node_parser "https://github.com/hhshanto/automated_moodle_scraper/blob/main/src/moodle_scraper/parser.py"
click node_prompt "https://github.com/hhshanto/automated_moodle_scraper/blob/main/src/moodle_scraper/prompt.txt"
click node_tools "https://github.com/hhshanto/automated_moodle_scraper/blob/main/src/moodle_scraper/tools.py"
click node_browser "https://github.com/hhshanto/automated_moodle_scraper/blob/main/src/moodle_scraper/browser.py"
click node_auth "https://github.com/hhshanto/automated_moodle_scraper/blob/main/src/moodle_scraper/auth.py"
click node_utils "https://github.com/hhshanto/automated_moodle_scraper/blob/main/src/moodle_scraper/utils.py"
click node_models "https://github.com/hhshanto/automated_moodle_scraper/blob/main/src/moodle_scraper/models.py"
click node_exporter "https://github.com/hhshanto/automated_moodle_scraper/blob/main/src/moodle_scraper/exporter.py"
click node_tests_auth "https://github.com/hhshanto/automated_moodle_scraper/blob/main/tests/test_auth.py"
click node_tests_parser "https://github.com/hhshanto/automated_moodle_scraper/blob/main/tests/test_parser.py"
click node_tests_azure "https://github.com/hhshanto/automated_moodle_scraper/blob/main/tests/test_azure_openai.py"
click node_readme "https://github.com/hhshanto/automated_moodle_scraper/blob/main/README.md"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main_py,node_mcp_server toneBlue
class node_tools,node_browser,node_auth,node_utils toneAmber
class node_agent,node_parser,node_prompt toneMint
class node_models,node_exporter toneRose
class node_tests_auth,node_tests_parser,node_tests_azure,node_readme toneNeutral
```

## Components

### Runtime & control

Entry points and alternate control planes for the scraper

- [main.py](https://github.com/hhshanto/automated_moodle_scraper/blob/main/main.py): entrypoint · Runtime switchboard for MCP server, agent mode, and login smoke test · `main.py`
- [MCP server](https://github.com/hhshanto/automated_moodle_scraper/blob/main/src/moodle_scraper/mcp_server.py): service · Exposes browser tools to external MCP-compatible clients · `src/moodle_scraper/mcp_server.py`

### Browser automation

Playwright session, auth, and shared browser actions

- [Tools](https://github.com/hhshanto/automated_moodle_scraper/blob/main/src/moodle_scraper/tools.py): browser actions · Single source of truth for browser-capable scraping actions · `src/moodle_scraper/tools.py`
- [Browser](https://github.com/hhshanto/automated_moodle_scraper/blob/main/src/moodle_scraper/browser.py): playwright session · Owns Playwright lifecycle, page state, screenshots, and shutdown · `src/moodle_scraper/browser.py`
- [Auth](https://github.com/hhshanto/automated_moodle_scraper/blob/main/src/moodle_scraper/auth.py): login flow · Handles Moodle login and DFN-AAI SSO boundary · `src/moodle_scraper/auth.py`
- [Utils](https://github.com/hhshanto/automated_moodle_scraper/blob/main/src/moodle_scraper/utils.py): cross-cutting · Logging, timestamps, and output path helpers · `src/moodle_scraper/utils.py`

### LLM orchestration

Standalone agent loop and Azure OpenAI integration

- [Agent loop](https://github.com/hhshanto/automated_moodle_scraper/blob/main/src/moodle_scraper/agent.py): orchestrator · Runs the standalone tool-calling loop against Azure OpenAI · `src/moodle_scraper/agent.py`
- [Azure client](https://github.com/hhshanto/automated_moodle_scraper/blob/main/src/moodle_scraper/parser.py): llm client · Wraps Azure OpenAI calls used by the agent loop · `src/moodle_scraper/parser.py`
- [Prompt](https://github.com/hhshanto/automated_moodle_scraper/blob/main/src/moodle_scraper/prompt.txt): instructions · Agent prompt and control instructions · `src/moodle_scraper/prompt.txt`

### Data & output

Models, parsing, and export boundaries

- [Models](https://github.com/hhshanto/automated_moodle_scraper/blob/main/src/moodle_scraper/models.py): data model · Structured quiz and course representations · `src/moodle_scraper/models.py`
- [Exporter](https://github.com/hhshanto/automated_moodle_scraper/blob/main/src/moodle_scraper/exporter.py): output · Writes XML, HTML, and screenshot artifacts · `src/moodle_scraper/exporter.py`

### Other components

- [Auth tests](https://github.com/hhshanto/automated_moodle_scraper/blob/main/tests/test_auth.py): test · Covers Moodle authentication and SSO behavior · `tests/test_auth.py`
- [Parser tests](https://github.com/hhshanto/automated_moodle_scraper/blob/main/tests/test_parser.py): test · Covers parsing and model boundary behavior · `tests/test_parser.py`
- [Azure tests](https://github.com/hhshanto/automated_moodle_scraper/blob/main/tests/test_azure_openai.py): test · Covers Azure OpenAI integration seam · `tests/test_azure_openai.py`
- [README](https://github.com/hhshanto/automated_moodle_scraper/blob/main/README.md): docs · Describes modes, architecture, and planned features · `README.md`

## Connections

- main.py → MCP server: default mode · Starts the MCP service
- main.py → Agent loop: --agent · Runs standalone agent mode
- main.py → Auth: --test-login · Runs manual login smoke test
- MCP server → Tools: exposes · Publishes the shared browser tool surface
- Agent loop → Azure client: calls · Requests tool decisions from Azure OpenAI
- Agent loop → Prompt: loads · Uses prompt instructions for the loop
- Agent loop → Tools: executes · Runs browser actions returned by the model
- Azure client → Tools: tool schema · Shapes calls around the shared tool contract
- Tools → Browser: controls · Uses the Playwright session for navigation and capture
- Tools → Auth: login · Invokes Moodle authentication when needed
- Tools → Utils: uses · Shares logging and filesystem helpers
- Browser → Utils: uses · Relies on common runtime helpers
- Auth → Browser: drives · Performs login through the browser session
- Azure client → Models: structures · Converts unstructured responses toward typed data
- Exporter → Models: serializes · Exports structured scraper results
- Exporter → Utils: writes via · Uses output-path and logging helpers
- Auth tests → Auth: validates · Exercises authentication boundary
- Parser tests → Azure client: validates · Exercises parsing boundary
- Azure tests → Azure client: validates · Exercises Azure client integration
- README → main.py: documents · Explains runtime modes and entry points
- README → Tools: documents · Describes the shared automation core
- README → Exporter: documents · Describes planned export pipeline