# cemreefe/coursemaker architecture

> Architecture diagram of the GitHub repository cemreefe/coursemaker, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cemreefe/coursemaker
- Repository: https://github.com/cemreefe/coursemaker

## Overview

##### Project type and purpose
- This is a command-line tool (an “open-source utility”) called Course Maker that automates the process of converting a raw text corpus into a sequence of language-learning sentences and then formatting them into a course.

##### File structure
- Top level scripts:
- preprocess_corpus.py — splits raw text into sentences and builds word-frequency CSVs
- sort_sentences.py — applies the frequency-based selection algorithm to rank sentences
- formatter.py — renders the ranked sentences into Markdown or HTML course pages
- add_translation.py — (future) injects sentence translations from a dictionary CSV
- Support directories:
- lib/stemmers/ru_stemmer — optional Russian stemming engine used by preprocess and sorting
- courses/ — example output courses (JSON and MD)
- media/ — graphs and images referenced in the README
- No front end or backend web service; everything is a local pipeline reading/writing CSV, JSON, MD, or HTML.

##### README for additional insights
- Describes a three-stage pipeline:
a) Preprocessing raw corpora into sentences & frequency tables
b) Greedy sentence sorting to maximize new-word coverage
c) Formatting output into human-readable materials
- Optional features: stemming (via lib/stemmers), dictionary-based translations, multiple output formats.

##### Main components and their interactions
a. Input layer:
- Raw text corpus file
- (Optional) Pre-tokenization rules, language alphabet flag
b. Preprocessing module (preprocess_corpus.py):
- Tokenizer, normalizer, optional stemmer hook → outputs sentences.csv + freqs.csv
c. Stemming library (lib/stemmers/ru_stemmer):
- Invoked by preprocess and sort modules when ‑stm ru is passed
d. Sorting module (sort_sentences.py):
- Reads sentences.csv & freqs.csv → applies greedy average-frequency algorithm → writes sorted_sentences.csv
e. Optional translation injector (add_translation.py):
- Takes sorted_sentences.csv + dictionary.csv → outputs enriched CSV with “translation” column
f. Formatter module (formatter.py):
- Reads final CSV → renders Markdown or HTML course materials → writes courses/<lang>_course.md or .html
g. Output layer: courses/, which can then be served or published.

##### Architectural patterns and design principles
- Pipeline pattern: clear stages chained by files
- Modular CLI commands with consistent flag conventions
- Pluggable components (stemming, translation) isolated under lib/ or separate scripts
- Separation of concerns: preprocessing, sorting logic, formatting/rendering

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Input Layer
    Raw["Raw Corpus Text"]:::artifact

    %% Preprocessing Module
    Pre["preprocess_corpus.py"]:::preproc
    Raw -->|"reads raw text"| Pre
    Pre -->|"produces sentences.csv"| SentencesCSV["sentences.csv"]:::artifact
    Pre -->|"produces freqs.csv"| FreqCSV["freqs.csv"]:::artifact
    Raw -.->|"optional -stm ru"| StemLib["ru_stemmer"]:::optional
    Pre -.->|"uses stemmer hook"| StemLib

    %% Stemming Library
    subgraph "Stemming Library (Russian)" 
        direction TB
        StemCore["stemmer.py"]:::optional
        StemDocs["readme.md"]:::optional
        StemSample["sample_1.txt.stemmed"]:::optional
    end
    StemLib --> StemCore
    StemLib --> StemDocs
    StemLib --> StemSample

    %% Sorting Module
    Sort["sort_sentences.py"]:::sorting
    SentencesCSV -->|"reads sentences.csv"| Sort
    FreqCSV -->|"reads freqs.csv"| Sort
    Sort -->|"writes sorted_sentences.csv"| SortedCSV["sorted_sentences.csv"]:::artifact
    Sort -.->|"optional -stm ru"| StemLib

    %% Translation Injector (Optional)
    Trans["add_translation.py"]:::optional
    SortedCSV -->|"reads sorted_sentences.csv"| Trans
    DictCSV["dictionary.csv"]:::artifact
    DictCSV -->|"-d dictionary.csv"| Trans
    Trans -->|"writes translated_sentences.csv"| TransCSV["translated_sentences.csv"]:::artifact

    %% Formatter Module
    Format["formatter.py"]:::formatting
    SortedCSV -->|"reads sorted_sentences.csv"| Format
    TransCSV -->|"reads translated_sentences.csv"| Format
    Format -->|"writes course.md"| CourseMD["courses/<lang>_course.md"]:::artifact
    Format -->|"writes course.html"| CourseHTML["courses/<lang>_course.html"]:::artifact

    %% Output Layer
    subgraph "Output Layer"
        direction TB
        CoursesDir["courses/"]:::artifact
        CourseJSON["russian_course.json"]:::artifact
        CourseMD
        CourseHTML
    end
    CourseMD --> CoursesDir
    CourseHTML --> CoursesDir
    CourseJSON --> CoursesDir

    %% External Dependencies
    subgraph "External Dependencies"
        direction TB
        PyRuntime["Python Runtime"]:::external
        ThirdParty["Third-Party Libs"]:::external
        OnlineAPI["Online Translation API (future)"]:::external
    end
    Pre --> PyRuntime
    Sort --> PyRuntime
    Format --> PyRuntime
    Pre --> ThirdParty
    Sort --> ThirdParty
    Trans --> OnlineAPI

    %% Legend
    subgraph Legend
        direction TB
        A1["Module (Blue)"]:::preproc
        A2["Module (Green)"]:::sorting
        A3["Module (Orange)"]:::formatting
        A4["Optional/Lib (Gray)"]:::optional
        A5["Data Artifact"]:::artifact
        A6["External"]:::external
    end

    %% Click Events
    click Pre "https://github.com/cemreefe/coursemaker/blob/main/preprocess_corpus.py"
    click Sort "https://github.com/cemreefe/coursemaker/blob/main/sort_sentences.py"
    click Format "https://github.com/cemreefe/coursemaker/blob/main/formatter.py"
    click Trans "https://github.com/cemreefe/coursemaker/blob/main/add_translation.py"
    click StemLib "https://github.com/cemreefe/coursemaker/tree/main/lib/stemmers/ru_stemmer/"
    click StemCore "https://github.com/cemreefe/coursemaker/blob/main/lib/stemmers/ru_stemmer/stemmer.py"
    click StemDocs "https://github.com/cemreefe/coursemaker/blob/main/lib/stemmers/ru_stemmer/readme.md"
    click StemSample "https://github.com/cemreefe/coursemaker/blob/main/lib/stemmers/ru_stemmer/samples/sample_1.txt.stemmed"
    click CoursesDir "https://github.com/cemreefe/coursemaker/tree/main/courses/"
    click CourseJSON "https://github.com/cemreefe/coursemaker/blob/main/courses/russian_course.json"
    click CourseMD "https://github.com/cemreefe/coursemaker/blob/main/courses/russian_course.md"
    click media "https://github.com/cemreefe/coursemaker/tree/main/media/"

    %% Styles
    classDef preproc fill:#D6E9FF,stroke:#4DA6FF,stroke-width:1px
    classDef sorting fill:#DFFFD6,stroke:#4DA64D,stroke-width:1px
    classDef formatting fill:#FFE5B4,stroke:#FFB24D,stroke-width:1px
    classDef optional fill:#ECECEC,stroke:#999999,stroke-width:1px
    classDef artifact fill:#FFF9C4,stroke:#E6D600,stroke-width:1px
    classDef external fill:#F0F0F0,stroke:#AAAAAA,stroke-width:1px,stroke-dasharray: 5 5
```
