How e235221/syllabusurl_to_markdown works
Project type and purpose
• This is a command-line Python tool whose sole responsibility is to fetch a set of university syllabus web pages (from tiglon.jim.u-ryukyu.ac.jp), parse their contents, and emit well-structured Markdown files.
• There is no web frontend or REST API; it’s a standalone “scrape + transform + write” utility.
File structure
• Top-level script: syllabus_url_to_md.py – contains the main entry point, CLI flags, interactive prompt, and orchestration logic.
• requirements.txt – lists selenium, possibly webdriver-manager, and other text-processing dependencies.
• Syllabus_venv/ – a Python virtual environment (isolated interpreter + installed libraries).
• syllabus_format_analysis.md & memo.md – design notes or format analyses (auxiliary documentation).
• .gitignore – ensures Syllabus_venv/, __pycache__/, and output directories are not checked in.
README for additional insights
• Technology stack: Python 3.12.4, Selenium + headless Chrome, re module for parsing, output Markdown.
• Execution modes: batch via command-line arguments vs. interactive prompt (asks for URL(s) and output directory).
• Output structure: Markdown sections (title, info table, bullet-list sections, links).
Main components and relationships
a. Input layer
• CLI parser / interactive prompt (argparse / input())
• Accepts one or more syllabus URLs + optional output directory
b. Scraping layer
• Selenium WebDriver
• Headless Chrome/Chromium controlled by the Python script
• Fetches dynamic content, waits for page load
c. Parsing layer
• Python’s re-based routines (inlined in the script or in helper functions)
• Extracts structured fields (course number, credits, schedule, etc.)
d. Transformation layer
• Assembles parsed data into Markdown text
• Renders tables, headings (##), lists, and link syntax
e. Output layer
• File I/O module
• Writes .md files under the specified directory (default: current working directory)
f. Environment & Dependencies
• Virtual environment (Syllabus_venv) isolating Python interpreter + installed packages
• requirements.txt ensures reproducible installs
Interactions:
• The CLI passes user inputs to the orchestration logic
• Orchestration invokes the Selenium driver → returns raw HTML/text
• Parsing routines consume HTML/text → produce structured Python objects
• Markdown generator consumes objects → yields text blobs
• File writer persists blobs → disk
Architectural patterns and design principles
• Single-process, sequential pipeline (no concurrency unless Selenium is used in parallel)
• Clear separation of concerns: input gathering → web scraping → parsing → formatting → writing
• Dependency isolation via virtual environment + requirements file
• Fail-fast: errors in scraping or parsing should abort with clear messages