How hhshanto/automated_moodle_scraper works
1. What this repo is
- This is a browser automation pipeline for Moodle, centered on a Playwright-driven browser session that can be controlled either by an external MCP client or by an internal Azure OpenAI agent loop.
- The architecture is split between tool definitions, browser/auth session management, LLM orchestration, and export/output.
2. Primary entry point
- main.py is the runtime switchboard.
- It supports three modes: default MCP server, --agent for the standalone Azure OpenAI tool-calling loop, and --test-login for a manual browser login smoke test.
- This means the repo is not a single workflow; it is a set of reusable capabilities exposed through different control planes.
3. Shared automation core
- src/moodle_scraper/tools.py is the central boundary for browser actions.
- The README states this is the “single source of truth” for tools, and both the MCP server and the standalone agent call into it.
- That design prevents duplicated logic between “LLM client controls browser” and “internal agent controls browser” paths.
4. Browser/session layer
- src/moodle_scraper/browser.py likely owns the Playwright browser lifecycle: launch, page context, screenshots, and shutdown.
- This is the execution substrate for all scraping actions and the main stateful boundary in the system.
- The current implemented tools indicate the browser layer already supports navigation, visible-page text extraction, screenshots, status checks, and closure.
5. Authentication boundary
- src/moodle_scraper/auth.py encapsulates Moodle login, including DFN-AAI SSO flow handling.
- This is a separate subsystem from generic browsing because it has site-specific selectors and identity-provider behavior.
- main.py --test-login exists specifically to validate this boundary without any LLM involvement.
6. LLM-driven orchestration
- src/moodle_scraper/agent.py implements the standalone agent loop: it sends the goal to Azure OpenAI, receives tool-call decisions, executes tools, and repeats until completion.
- src/moodle_scraper/parser.py is the Azure OpenAI client wrapper used by that loop.
- src/moodle_scraper/prompt.txt likely contains the agent’s system prompt or instructions, making prompt behavior part of the orchestration layer rather than the tool layer.
7. MCP server integration
- src/moodle_scraper/mcp_server.py exposes the same tool surface through Model Context Protocol.
- This turns the scraper into a service that external LLM clients such as Claude Desktop or VS Code Copilot can control.
- Architecturally, MCP is just an alternate control interface over the same underlying automation core.
8. Data model and parsing layer
- src/moodle_scraper/models.py is intended for structured quiz/course data, likely via Pydantic.
- src/moodle_scraper/parser.py sits at the boundary where unstructured browser/page content is turned into model-driven, LLM-assisted structured output.
- In the current repo state, this is more scaffolded than fully realized, since the README marks quiz scraping/export as planned.
9. Export/output subsystem
- src/moodle_scraper/exporter.py is the output boundary for XML, HTML, and screenshots.
- Screenshots are already part of the implemented workflow and are stored under output/screenshots/.
- The planned XML/HTML exports indicate a future stage where scraped Moodle quiz data becomes portable artifacts, not just browser state.
10. Utilities and cross-cutting concerns
- src/moodle_scraper/utils.py centralizes logging, timestamps, and output path handling.
- .env-driven configuration controls Moodle credentials, Azure OpenAI settings, headless mode, and output location.
- This keeps environment-specific concerns outside the browser and agent logic.
11. External systems and runtime dependencies
- The browser automation depends on Playwright + Chromium.
- Authentication targets a Moodle instance with DFN-AAI SSO.
- The agent mode depends on Azure OpenAI GPT-4o, specifically for tool-calling and vision-enabled interaction.
- The MCP mode depends on MCP-compatible clients rather than a direct OpenAI loop.
12. Current maturity of the system
- The repository is partly operational and partly scaffolded.
- Implemented today: login, navigation, screenshots, page text extraction, status checks, browser shutdown, MCP exposure, and Azure OpenAI orchestration.
- Planned next: course discovery, quiz listing, quiz scraping, and export to XML/HTML.
13. Test coverage boundary
- tests/test_auth.py, tests/test_parser.py, and tests/test_azure_openai.py validate the two most fragile seams: Moodle authentication and Azure OpenAI integration.
- This reflects the architecture’s two biggest external dependencies: identity-provider login and model/API interaction.
Main entry points: tools.py Tools · parser.py Azure client · agent.py Agent loop · main.py main.py
Components 15
Runtime & control
Entry points and alternate control planes for the scraper
- main.py · Runtime switchboard for MCP server, agent mode, and login smoke test
main.py - MCP server · Exposes browser tools to external MCP-compatible clients
src/moodle_scraper/mcp_server.py
Browser automation
Playwright session, auth, and shared browser actions
- Tools · Single source of truth for browser-capable scraping actions
src/moodle_scraper/tools.py - Browser · Owns Playwright lifecycle, page state, screenshots, and shutdown
src/moodle_scraper/browser.py - Auth · Handles Moodle login and DFN-AAI SSO boundary
src/moodle_scraper/auth.py - Utils · Logging, timestamps, and output path helpers
src/moodle_scraper/utils.py
LLM orchestration
Standalone agent loop and Azure OpenAI integration
- Agent loop · Runs the standalone tool-calling loop against Azure OpenAI
src/moodle_scraper/agent.py - Azure client · Wraps Azure OpenAI calls used by the agent loop
src/moodle_scraper/parser.py - Prompt · Agent prompt and control instructions
src/moodle_scraper/prompt.txt
Data & output
Models, parsing, and export boundaries
Other components
- Auth tests · Covers Moodle authentication and SSO behavior
tests/test_auth.py - Parser tests · Covers parsing and model boundary behavior
tests/test_parser.py - Azure tests · Covers Azure OpenAI integration seam
tests/test_azure_openai.py - README · Describes modes, architecture, and planned features
README.md
Connections 22
- main.py → MCP server: default mode · Starts the MCP service
- main.py → Agent loop: --agent · Runs standalone agent mode
- main.py → Auth: --test-login · Runs manual login smoke test
- MCP server → Tools: exposes · Publishes the shared browser tool surface
- Agent loop → Azure client: calls · Requests tool decisions from Azure OpenAI
- Agent loop → Prompt: loads · Uses prompt instructions for the loop
- Agent loop → Tools: executes · Runs browser actions returned by the model
- Azure client → Tools: tool schema · Shapes calls around the shared tool contract
- Tools → Browser: controls · Uses the Playwright session for navigation and capture
- Tools → Auth: login · Invokes Moodle authentication when needed
- Tools → Utils: uses · Shares logging and filesystem helpers
- Browser → Utils: uses · Relies on common runtime helpers
- Auth → Browser: drives · Performs login through the browser session
- Azure client → Models: structures · Converts unstructured responses toward typed data
- Exporter → Models: serializes · Exports structured scraper results
- Exporter → Utils: writes via · Uses output-path and logging helpers
- Auth tests → Auth: validates · Exercises authentication boundary
- Parser tests → Azure client: validates · Exercises parsing boundary
- Azure tests → Azure client: validates · Exercises Azure client integration
- README → main.py: documents · Explains runtime modes and entry points
- README → Tools: documents · Describes the shared automation core
- README → Exporter: documents · Describes planned export pipeline