# e235221/syllabusurl_to_markdown architecture

> Architecture diagram of the GitHub repository e235221/syllabusurl_to_markdown, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/e235221/syllabusurl_to_markdown
- Repository: https://github.com/e235221/syllabusurl_to_markdown

## Overview

##### Project type and purpose
- This is a command-line Python tool whose sole responsibility is to fetch a set of university syllabus web pages (from tiglon.jim.u-ryukyu.ac.jp), parse their contents, and emit well-structured Markdown files.
- There is no web frontend or REST API; it’s a standalone “scrape + transform + write” utility.

##### File structure
- Top-level script: `syllabus_url_to_md.py` – contains the main entry point, CLI flags, interactive prompt, and orchestration logic.
- `requirements.txt` – lists `selenium`, possibly `webdriver-manager`, and other text-processing dependencies.
- `Syllabus_venv/` – a Python virtual environment (isolated interpreter + installed libraries).
- `syllabus_format_analysis.md` & `memo.md` – design notes or format analyses (auxiliary documentation).
- `.gitignore` – ensures `Syllabus_venv/`, `__pycache__/`, and output directories are not checked in.

##### README for additional insights
- Technology stack: Python 3.12.4, Selenium + headless Chrome, `re` module for parsing, output Markdown.
- Execution modes: batch via command-line arguments vs. interactive prompt (asks for URL(s) and output directory).
- Output structure: Markdown sections (title, info table, bullet-list sections, links).

##### Main components and relationships
a. Input layer
- CLI parser / interactive prompt (argparse / input())
- Accepts one or more syllabus URLs + optional output directory
b. Scraping layer
- Selenium WebDriver
- Headless Chrome/Chromium controlled by the Python script
- Fetches dynamic content, waits for page load
c. Parsing layer
- Python’s `re`-based routines (inlined in the script or in helper functions)
- Extracts structured fields (course number, credits, schedule, etc.)
d. Transformation layer
- Assembles parsed data into Markdown text
- Renders tables, headings (`##`), lists, and link syntax
e. Output layer
- File I/O module
- Writes `.md` files under the specified directory (default: current working directory)
f. Environment & Dependencies
- Virtual environment (`Syllabus_venv`) isolating Python interpreter + installed packages
- `requirements.txt` ensures reproducible installs

Interactions:
- The CLI passes user inputs to the orchestration logic
- Orchestration invokes the Selenium driver → returns raw HTML/text
- Parsing routines consume HTML/text → produce structured Python objects
- Markdown generator consumes objects → yields text blobs
- File writer persists blobs → disk

##### Architectural patterns and design principles
- Single-process, sequential pipeline (no concurrency unless Selenium is used in parallel)
- Clear separation of concerns: input gathering → web scraping → parsing → formatting → writing
- Dependency isolation via virtual environment + requirements file
- Fail-fast: errors in scraping or parsing should abort with clear messages

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External System
    University["University Syllabus System"]:::external

    %% Environment Boundary
    subgraph "Environment: Syllabus_venv" 
        style Environment stroke-dasharray: 5 5
        CLI["Argparse / Interactive Prompt"]:::runtime
        Scraper["Selenium WebDriver / ChromeDriver"]:::runtime
        Parser["Regex Parser"]:::runtime
        Transformer["Markdown Generator"]:::runtime
        Writer["File Writer"]:::runtime
    end

    %% Storage / Output
    docsFS["docs/<course>.md files"]:::storage

    %% Configuration & VCS
    requirements["requirements.txt"]:::config
    gitignore[".gitignore"]:::config
    VenvBoundary["Syllabus_venv/"]:::config

    %% Documentation
    README["README.md"]:::doc
    memo["memo.md"]:::doc
    analysis["syllabus_format_analysis.md"]:::doc

    %% Data Flow
    University -->|"page HTML"| Scraper
    CLI -->|"URL list"| Scraper
    Scraper -->|"page HTML"| Parser
    Parser -->|"parsed fields"| Transformer
    Transformer -->|"Markdown text"| Writer
    Writer -->|".md files"| docsFS

    %% Config interactions
    requirements -->|"installed packages"| VenvBoundary
    gitignore -->|"VCS exclusions"| VenvBoundary

    %% Docs references
    README --- CLI
    memo --- Parser
    analysis --- Transformer

    %% Click Events
    click CLI "https://github.com/e235221/syllabusurl_to_markdown/blob/main/syllabus_url_to_md.py"
    click Scraper "https://github.com/e235221/syllabusurl_to_markdown/blob/main/syllabus_url_to_md.py"
    click Parser "https://github.com/e235221/syllabusurl_to_markdown/blob/main/syllabus_url_to_md.py"
    click Transformer "https://github.com/e235221/syllabusurl_to_markdown/blob/main/syllabus_url_to_md.py"
    click Writer "https://github.com/e235221/syllabusurl_to_markdown/blob/main/syllabus_url_to_md.py"
    click requirements "https://github.com/e235221/syllabusurl_to_markdown/blob/main/requirements.txt"
    click gitignore "https://github.com/e235221/syllabusurl_to_markdown/blob/main/.gitignore"
    click VenvBoundary "https://github.com/e235221/syllabusurl_to_markdown/tree/main/Syllabus_venv/"
    click memo "https://github.com/e235221/syllabusurl_to_markdown/blob/main/memo.md"
    click analysis "https://github.com/e235221/syllabusurl_to_markdown/blob/main/syllabus_format_analysis.md"
    click README "https://github.com/e235221/syllabusurl_to_markdown/blob/main/README.md"

    %% Styles
    classDef external fill:#cccccc,stroke:#666666,color:#333333
    classDef runtime fill:#cce5ff,stroke:#3399ff,color:#0b2e59
    classDef storage fill:#d4edda,stroke:#28a745,color:#155724
    classDef config fill:#fff3cd,stroke:#e0a800,color:#856404
    classDef doc fill:#fefefe,stroke:#aaaaaa,color:#444444
```
