How cemreefe/coursemaker works
Project type and purpose
• This is a command-line tool (an “open-source utility”) called Course Maker that automates the process of converting a raw text corpus into a sequence of language-learning sentences and then formatting them into a course.
File structure
• Top level scripts:
• preprocess_corpus.py — splits raw text into sentences and builds word-frequency CSVs
• sort_sentences.py — applies the frequency-based selection algorithm to rank sentences
• formatter.py — renders the ranked sentences into Markdown or HTML course pages
• add_translation.py — (future) injects sentence translations from a dictionary CSV
• Support directories:
• lib/stemmers/ru_stemmer — optional Russian stemming engine used by preprocess and sorting
• courses/ — example output courses (JSON and MD)
• media/ — graphs and images referenced in the README
• No front end or backend web service; everything is a local pipeline reading/writing CSV, JSON, MD, or HTML.
README for additional insights
• Describes a three-stage pipeline:
a) Preprocessing raw corpora into sentences & frequency tables
b) Greedy sentence sorting to maximize new-word coverage
c) Formatting output into human-readable materials
• Optional features: stemming (via lib/stemmers), dictionary-based translations, multiple output formats.
Main components and their interactions
a. Input layer:
• Raw text corpus file
• (Optional) Pre-tokenization rules, language alphabet flag
b. Preprocessing module (preprocess_corpus.py):
• Tokenizer, normalizer, optional stemmer hook → outputs sentences.csv + freqs.csv
c. Stemming library (lib/stemmers/ru_stemmer):
• Invoked by preprocess and sort modules when ‑stm ru is passed
d. Sorting module (sort_sentences.py):
• Reads sentences.csv & freqs.csv → applies greedy average-frequency algorithm → writes sorted_sentences.csv
e. Optional translation injector (add_translation.py):
• Takes sorted_sentences.csv + dictionary.csv → outputs enriched CSV with “translation” column
f. Formatter module (formatter.py):
• Reads final CSV → renders Markdown or HTML course materials → writes courses/<lang>_course.md or .html
g. Output layer: courses/, which can then be served or published.
Architectural patterns and design principles
• Pipeline pattern: clear stages chained by files
• Modular CLI commands with consistent flag conventions
• Pluggable components (stemming, translation) isolated under lib/ or separate scripts
• Separation of concerns: preprocessing, sorting logic, formatting/rendering