How harsh199323/lingomed-dev works
Project Type and Purpose
• LingoMed is an open-source, Python-based text-processing tool (a CLI application) that:
• Crawls German medical content from Wikipedia
• Detects medical vocabulary and grammatical patterns
• Generates language-learning exercises in YAML format
High-Level Architecture Components
Primary building blocks:
A. User Interface (CLI) / Entry Point
• src/main.py
B. Corpus Crawling Module
• src/corpus_crawler.py
• External dependency: Wikipedia-API + BeautifulSoup + requests
C. Data Storage
• Raw corpus files under data/corpora/*.txt
D. Sentence Extraction Module
• src/sentence_extractor.py
• Uses spaCy (German model) + pattern definitions from config/pattern.yml
E. Configuration Store
• config/pattern.yml (medical_terms + grammar_patterns)
F. Output Storage
• YAML files under results/*.yml
G. Tests (for completeness)
• tests/test_sentence_extractor.py
Relationships and Data Flow
• CLI (main.py) orchestrates two sequential steps:
1. Crawl → Module calls Wikipedia-API, writes combined corpus to data/corpus.txt
2. Extract → Reads corpus, loads patterns from config/pattern.yml, applies NLP, writes results to results/*.yml
• External Service: Wikipedia
• Libraries: spaCy (pattern matching), pyyaml, tqdm
• Error Handling & Logging flows alongside both modules
Architectural Patterns & Design Principles
• Pipeline Pattern: A two-stage pipeline (crawling → extraction)
• Separation of Concerns:
• Crawling logic decoupled from extraction logic
• Configuration externalized to YAML
• Extensibility Points:
• New grammar patterns or terms by editing pattern.yml
• Replace spaCy with stanza in future
Detailed Element List
• Components:
• main.py (CLI)
• CorpusCrawler class
• SentenceExtractor class
• pattern.yml (config)
• data/corpora/* (raw corpus)
• results/*.yml (output)
• External: Wikipedia API, spaCy model
• Interactions:
• Initialization → load config → start crawl
• Crawl → write combined_corpus.txt
• Extraction → load spaCy model + config → pattern matching → YAML generation
• Dependencies:
• requests, BeautifulSoup4, Wikipedia-API
• spaCy (de_core_news_sm)
• pyyaml, tqdm