How shingzhanho/verbe-conjugaison-academie-francaise works
Project type and purpose
• This is an open-source Python CLI tool whose sole purpose is to crawl the Académie Française (AF) website, parse verb conjugation tables, and produce two formats of release artifacts: JSON files (verbs.json / verbs.min.json) and an optional SQLite3 database (verbs.db).
High-level architecture/components
a. CLI / Entry Point
• crawler.py is the main script invoked by the user. It parses command-line arguments, sets options (e.g. overwrite cookie, user-agent, cache behavior, verbose), and orchestrates the pipeline.
b. Configuration & Globals
• global_vars.py holds default constants (endpoints, file paths, default delays, retry counts) and is imported wherever these defaults are needed.
c. HTTP Fetcher (Crawler module)
• Within crawler.py (and possibly core.py), HTTP requests are made to the AF website, honoring delay and retry options.
• HTML pages are cached on disk to avoid repeated downloads (if IGNORE_CACHE is false).
d. Parser module
• conjugation_parser.py takes cached HTML files and extracts conjugation data (voices, moods, tenses, persons). It produces an in-memory data structure (e.g. nested dicts).
e. Data Aggregator / Serializer
• After parsing, data is collated into JSON structures and written out as verbs.json (pretty) and verbs.min.json (minified).
f. Extension plugin: SQLite generator
• extensions/db.py implements a plugin that, when enabled (-E:GEN-SQLITE3), takes the same in-memory data and writes it into an SQLite3 database (verbs.db) with three tables per voice and standardized column names.
g. Logging
• log.py provides logging wrappers; all modules log progress, errors, and stats here.
h. Release artifacts
• Output files (verbs.json, verbs.min.json, verbs.db) live in the project root after the pipeline completes.
i. External service
• Académie Française website: the source of HTML pages.
Component interactions and data flow
• User invokes crawler.py → CLI parser sets up global_vars and log → calls Core pipeline in core.py.
• Core → HTTP Fetcher (requests to AF) → caches HTML locally.
• Cached HTML → Parser (conjugation_parser.py) → builds intermediate data model.
• Data model → JSON Serializer → writes verbs.json + verbs.min.json.
• If SQLite extension enabled → Data model → extensions/db.py → writes verbs.db.
• All along, log.py captures events; errors may trigger retries or abort.
Architectural patterns and design principles
• Pipeline pattern: distinct stages (fetch → parse → serialize → optional DB).
• Plugin/Extension pattern: clean separation between core JSON output and optional SQLite generation.
• Single Responsibility: each module (crawler, parser, db extension, logger) has a clear role.
Recommended elements
• CLI / crawler.py
• core.py (orchestrator)
• HTTP Fetcher
• Parser (conjugation_parser.py)
• JSON Serializer
• SQLite Extension (extensions/db.py)
• Logger (log.py)
• Config (global_vars.py)
• External AF Website
• Output artifacts (files)
• CLI → Orchestrator → Fetcher → Cache → Parser → Serializer → JSON files
• Serializer → (optional) DB Extension → verbs.db
• All modules → Logger
• CLI/Orchestrator → Config
• Fetcher ↔ AF Website
• Config & Logging (side modules, lighter shade)
Final advice
• Be exhaustive: include every Python script as a component, even if trivial.
• Cross-cutting concerns (config, logging) as separate side modules.
• Arrowheads clearly indicate direction of data or control flow.
• Don’t over-architect: keep it as simple pipeline stages plus optional extension.