# shingzhanho/verbe-conjugaison-academie-francaise architecture

> Architecture diagram of the GitHub repository shingzhanho/verbe-conjugaison-academie-francaise, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shingzhanho/verbe-conjugaison-academie-francaise
- Repository: https://github.com/shingzhanho/verbe-conjugaison-academie-francaise

## Overview

##### Project type and purpose
- This is an open-source Python CLI tool whose sole purpose is to crawl the Académie Française (AF) website, parse verb conjugation tables, and produce two formats of release artifacts: JSON files (`verbs.json` / `verbs.min.json`) and an optional SQLite3 database (`verbs.db`).

##### High-level architecture/components
a. CLI / Entry Point
- `crawler.py` is the main script invoked by the user. It parses command-line arguments, sets options (e.g. overwrite cookie, user-agent, cache behavior, verbose), and orchestrates the pipeline.
b. Configuration & Globals
- `global_vars.py` holds default constants (endpoints, file paths, default delays, retry counts) and is imported wherever these defaults are needed.
c. HTTP Fetcher (Crawler module)
- Within `crawler.py` (and possibly `core.py`), HTTP requests are made to the AF website, honoring delay and retry options.
- HTML pages are cached on disk to avoid repeated downloads (if `IGNORE_CACHE` is false).
d. Parser module
- `conjugation_parser.py` takes cached HTML files and extracts conjugation data (voices, moods, tenses, persons). It produces an in-memory data structure (e.g. nested dicts).
e. Data Aggregator / Serializer
- After parsing, data is collated into JSON structures and written out as `verbs.json` (pretty) and `verbs.min.json` (minified).
f. Extension plugin: SQLite generator
- `extensions/db.py` implements a plugin that, when enabled (`-E:GEN-SQLITE3`), takes the same in-memory data and writes it into an SQLite3 database (`verbs.db`) with three tables per voice and standardized column names.
g. Logging
- `log.py` provides logging wrappers; all modules log progress, errors, and stats here.
h. Release artifacts
- Output files (`verbs.json`, `verbs.min.json`, `verbs.db`) live in the project root after the pipeline completes.
i. External service
- Académie Française website: the source of HTML pages.

##### Component interactions and data flow
- User invokes `crawler.py` → CLI parser sets up `global_vars` and `log` → calls Core pipeline in `core.py`.
- Core → HTTP Fetcher (requests to AF) → caches HTML locally.
- Cached HTML → Parser (`conjugation_parser.py`) → builds intermediate data model.
- Data model → JSON Serializer → writes `verbs.json` + `verbs.min.json`.
- If SQLite extension enabled → Data model → `extensions/db.py` → writes `verbs.db`.
- All along, `log.py` captures events; errors may trigger retries or abort.

##### Architectural patterns and design principles
- Pipeline pattern: distinct stages (fetch → parse → serialize → optional DB).
- Plugin/Extension pattern: clean separation between core JSON output and optional SQLite generation.
- Single Responsibility: each module (crawler, parser, db extension, logger) has a clear role.

##### Recommended elements
- CLI / `crawler.py`
- `core.py` (orchestrator)
- HTTP Fetcher
- Parser (`conjugation_parser.py`)
- JSON Serializer
- SQLite Extension (`extensions/db.py`)
- Logger (`log.py`)
- Config (`global_vars.py`)
- External AF Website
- Output artifacts (files)
- CLI → Orchestrator → Fetcher → Cache → Parser → Serializer → JSON files
- Serializer → (optional) DB Extension → `verbs.db`
- All modules → Logger
- CLI/Orchestrator → Config
- Fetcher ↔ AF Website
- Config & Logging (side modules, lighter shade)

##### Final advice
- Be exhaustive: include every Python script as a component, even if trivial.
- Cross-cutting concerns (config, logging) as separate side modules.
- Arrowheads clearly indicate direction of data or control flow.
- Don’t over-architect: keep it as simple pipeline stages plus optional extension.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Core components
    CLI["CLI: crawler.py"]:::core
    Orchestrator["Orchestrator: core.py"]:::core
    Fetcher["HTTP Fetcher\n(crawler.py/core.py)"]:::core
    Parser["Parser\nconjugation_parser.py"]:::core
    Serializer["JSON Serializer\n(core.py)"]:::core

    %% Extension plugin
    DBExt["SQLite Extension\nextensions/db.py"]:::plugin

    %% Storage components
    Cache["HTML Cache\n(folder)"]:::storage

    %% Side modules
    Config["Config\nglobal_vars.py"]:::config
    Logger["Logger\nlog.py"]:::logging

    %% External service
    AF["Académie Française Website"]:::external

    %% Release artifacts
    subgraph "Release Artifacts"
        JSON1["verbs.json"]:::storage
        JSON2["verbs.min.json"]:::storage
        DBFile["verbs.db"]:::storage
    end

    %% Data/control flow
    CLI --> Orchestrator
    CLI --> Config
    Orchestrator --> Fetcher
    Fetcher --> AF
    AF --> Fetcher
    Fetcher --> Cache
    Cache --> Parser
    Parser --> Serializer
    Serializer --> JSON1
    Serializer --> JSON2
    Serializer --> DBExt
    DBExt --> DBFile

    %% Cross-cutting logging
    CLI --> Logger
    Orchestrator --> Logger
    Fetcher --> Logger
    Parser --> Logger
    Serializer --> Logger
    DBExt --> Logger

    %% Click events
    click CLI "https://github.com/shingzhanho/verbe-conjugaison-academie-francaise/blob/master/crawler.py"
    click Orchestrator "https://github.com/shingzhanho/verbe-conjugaison-academie-francaise/blob/master/core.py"
    click Config "https://github.com/shingzhanho/verbe-conjugaison-academie-francaise/blob/master/global_vars.py"
    click Fetcher "https://github.com/shingzhanho/verbe-conjugaison-academie-francaise/blob/master/crawler.py"
    click Parser "https://github.com/shingzhanho/verbe-conjugaison-academie-francaise/blob/master/conjugation_parser.py"
    click Serializer "https://github.com/shingzhanho/verbe-conjugaison-academie-francaise/blob/master/core.py"
    click DBExt "https://github.com/shingzhanho/verbe-conjugaison-academie-francaise/blob/master/extensions/db.py"
    click Logger "https://github.com/shingzhanho/verbe-conjugaison-academie-francaise/blob/master/log.py"

    %% Styles
    classDef core fill:#a2d2ff,stroke:#000,stroke-width:1px
    classDef plugin fill:#cdb4db,stroke:#000,stroke-width:1px
    classDef storage fill:#ade8f4,stroke:#000,stroke-width:1px
    classDef external fill:#ffafcc,stroke:#000,stroke-width:1px
    classDef config fill:#ffc6ff,stroke:#000,stroke-width:1px
    classDef logging fill:#bde0fe,stroke:#000,stroke-width:1px
```
