# harsh199323/lingomed-dev architecture

> Architecture diagram of the GitHub repository harsh199323/lingomed-dev, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/harsh199323/lingomed-dev
- Repository: https://github.com/harsh199323/lingomed-dev

## Overview

##### Project Type and Purpose
- LingoMed is an open-source, Python-based **text-processing tool** (a CLI application) that:
- Crawls German medical content from Wikipedia
- Detects medical vocabulary and grammatical patterns
- Generates language-learning exercises in YAML format

##### High-Level Architecture Components
Primary building blocks:
A. **User Interface (CLI) / Entry Point**
- `src/main.py`
B. **Corpus Crawling Module**
- `src/corpus_crawler.py`
- External dependency: Wikipedia-API + BeautifulSoup + requests
C. **Data Storage**
- Raw corpus files under `data/corpora/*.txt`
D. **Sentence Extraction Module**
- `src/sentence_extractor.py`
- Uses spaCy (German model) + pattern definitions from `config/pattern.yml`
E. **Configuration Store**
- `config/pattern.yml` (medical_terms + grammar_patterns)
F. **Output Storage**
- YAML files under `results/*.yml`
G. **Tests** (for completeness)
- `tests/test_sentence_extractor.py`

##### Relationships and Data Flow
- CLI (`main.py`) orchestrates two sequential steps:
1. **Crawl** → Module calls Wikipedia-API, writes combined corpus to `data/corpus.txt`
2. **Extract** → Reads corpus, loads patterns from `config/pattern.yml`, applies NLP, writes results to `results/*.yml`
- **External Service**: Wikipedia
- **Libraries**: spaCy (pattern matching), pyyaml, tqdm
- **Error Handling & Logging** flows alongside both modules

##### Architectural Patterns & Design Principles
- **Pipeline Pattern**: A two-stage pipeline (crawling → extraction)
- **Separation of Concerns**:
- Crawling logic decoupled from extraction logic
- Configuration externalized to YAML
- **Extensibility Points**:
- New grammar patterns or terms by editing `pattern.yml`
- Replace spaCy with stanza in future

##### Detailed Element List
- Components:
- main.py (CLI)
- CorpusCrawler class
- SentenceExtractor class
- pattern.yml (config)
- data/corpora/* (raw corpus)
- results/*.yml (output)
- External: Wikipedia API, spaCy model
- Interactions:
- Initialization → load config → start crawl
- Crawl → write combined_corpus.txt
- Extraction → load spaCy model + config → pattern matching → YAML generation
- Dependencies:
- requests, BeautifulSoup4, Wikipedia-API
- spaCy (de_core_news_sm)
- pyyaml, tqdm

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Systems
    Wikipedia["Wikipedia API"]:::external
    spaCy["spaCy (de_core_news_sm)"]:::external

    %% Configuration
    PatternConfig["pattern.yml"]:::config

    %% Core Modules
    CLI["main.py"]:::internal
    CorpusCrawler["CorpusCrawler\n(src/corpus_crawler.py)"]:::internal
    SentenceExtractor["SentenceExtractor\n(src/sentence_extractor.py)"]:::internal

    %% Data Storage
    subgraph "Raw & Combined Corpus Storage" 
        direction TB
        RawCorpus["data/corpora/"]:::storage
        CombinedCorpus["data/corpus.txt"]:::storage
    end

    subgraph "Output Storage"
        direction TB
        ExtractedExercises["extracted_exercises.yml"]:::storage
        ExtractedSentences["extracted_sentences.yml"]:::storage
    end

    %% Tests
    TestExtractor["test_sentence_extractor.py"]:::test

    %% Data Flow
    Wikipedia -->|HTTPrequests| CorpusCrawler
    CorpusCrawler -->|writesrawtexts| RawCorpus
    CorpusCrawler -->|writescombined| CombinedCorpus
    CLI -->|startscrawl| CorpusCrawler
    CLI -->|startsextract| SentenceExtractor
    PatternConfig --> SentenceExtractor
    CombinedCorpus -->|readscorpus.txt| SentenceExtractor
    spaCy --> SentenceExtractor
    SentenceExtractor -->|YAMLoutput| ExtractedExercises
    SentenceExtractor -->|YAMLoutput| ExtractedSentences
    TestExtractor -->|tests| SentenceExtractor

    %% Click Events
    click CLI "https://github.com/harsh199323/lingomed-dev/blob/main/src/main.py"
    click CorpusCrawler "https://github.com/harsh199323/lingomed-dev/blob/main/src/corpus_crawler.py"
    click RawCorpus "https://github.com/harsh199323/lingomed-dev/tree/main/data/corpora/"
    click CombinedCorpus "https://github.com/harsh199323/lingomed-dev/blob/main/data/corpus.txt"
    click SentenceExtractor "https://github.com/harsh199323/lingomed-dev/blob/main/src/sentence_extractor.py"
    click PatternConfig "https://github.com/harsh199323/lingomed-dev/blob/main/config/pattern.yml"
    click ExtractedExercises "https://github.com/harsh199323/lingomed-dev/blob/main/results/extracted_exercises.yml"
    click ExtractedSentences "https://github.com/harsh199323/lingomed-dev/blob/main/results/extracted_sentences.yml"
    click TestExtractor "https://github.com/harsh199323/lingomed-dev/blob/main/tests/test_sentence_extractor.py"

    %% Styles
    classDef internal fill:#D0E8FF,stroke:#0366d6,color:#0366d6;
    classDef storage fill:#E6FFEA,stroke:#22863A,color:#22863A;
    classDef external fill:#FFE5B4,stroke:#D97706,color:#D97706;
    classDef config fill:#FFF5B1,stroke:#DBAB09,color:#DBAB09;
    classDef test fill:#F0F0F0,stroke:#6A737D,color:#6A737D;
```
