# zeeguu/api architecture

> Architecture diagram of the GitHub repository zeeguu/api, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zeeguu/api
- Repository: https://github.com/zeeguu/api

## Overview

1. **System shape**
   This repo is a Python backend monorepo centered on the Zeeguu API, with two auxiliary NLP services (`asr_service`, `stanza_service`) and a large set of maintenance/ETL scripts under `tools/` and `zeeguu/operations/`. The main runtime is the API service; the other parts support content ingestion, model updates, and offline batch work.

2. **API boundary**
   The entry point for the product backend is `zeeguu/api/app.py`, with route modules under `zeeguu/api/endpoints/`. These endpoints are organized by domain: accounts, articles, bookmarks/words, sessions, exercises, search, translation, notifications, stats, topics, video/audio, and a dedicated `teacher_dashboard/` subtree. `zeeguu/api/utils/` contains cross-cutting concerns such as session handling, rate limiting, security checks, background jobs, and JSON/abort wrappers.

3. **Core domain layer**
   Most business logic lives in `zeeguu/core/`. This layer is split by capability rather than by transport: account management, content retrieval and cleaning, bookmark operations, exercises, scheduling, user statistics, friendships, leaderboards, badges, language difficulty, NLP, LLM services, translation services, audio lessons, verbal flashcards, and content recommender logic. The API layer should be treated as thin orchestration over this core layer.

4. **Persistence model**
   The persistence boundary is represented by `zeeguu/core/model/`, which contains the domain entities for users, articles, bookmarks, meanings, sessions, topics, feeds, videos, audio lessons, badges, notifications, cohorts, and many join/association tables. Database evolution is managed via the many SQL migration files in `tools/migrations/`, which indicates a schema-driven MySQL-backed architecture with frequent evolutionary changes.

5. **Reading/content ingestion pipeline**
   Content enters the system through feed and crawler components in `zeeguu/core/feed_handler/`, `zeeguu/core/content_retriever/`, and the crawler scripts under `tools/crawler/` and `zeeguu/operations/crawler/`. Raw articles/videos are downloaded, parsed, cleaned, quality-filtered, and then stored as content entities. The pipeline also includes content quality detection (advertorials, disturbing content, paywalls) and tokenization/normalization before the material is used in user-facing flows.

6. **Search and recommendation**
   Search and recommendation are backed by Elastic-related modules in `zeeguu/core/elastic/`, `zeeguu/core/content_recommender/`, and `zeeguu/core/semantic_search/`. These components build/query indexes, support semantic retrieval, and drive the “recommend texts in the zone of proximal development” use case described in the README. There is a clear flow from ingested content into search/indexing, then into personalized recommendations.

7. **NLP and language understanding**
   Language processing is a major subsystem: tokenization (`zeeguu/core/tokenization/`), multi-word expression detection (`zeeguu/core/mwe/`), linguistic pipelines (`zeeguu/core/nlp_pipeline/`), translation services, and language difficulty estimation (`zeeguu/core/language/`). The code supports multiple strategies for difficulty assessment, including frequency-based, Flesch-Kincaid, and ML-based CEFR classification. This layer feeds both content annotation and learner modeling.

8. **External AI/NLP services**
   Several modules are explicit adapters to external services: `zeeguu/core/llm_services/` wraps providers such as Anthropic and DeepSeek for simplification, example generation, grammar correction, and validation; `zeeguu/core/translation_services/` includes Deepl and Azure alignment; `zeeguu/core/audio_lessons/azure_voice_synthesizer.py` drives TTS. These integrations materially shape the architecture because lesson generation and text simplification depend on third-party AI services.

9. **Audio lesson pipeline**
   Audio lessons are a distinct product subsystem. `zeeguu/core/audio_lessons/` contains lesson generation, script building, suggestion validation, word selection, voice config, and ASR client integration. The standalone `asr_service/` and `stanza_service/` directories are deployed as separate containers, suggesting they are called over HTTP by the core service for speech recognition and NLP model execution.

10. **Learner model and activity tracking**
   The app tracks reading, browsing, listening, watching, exercise sessions, bookmarks, and user word interactions through dedicated session and statistics modules. `zeeguu/core/user_activity_hooks/`, `zeeguu/core/user_statistics/`, `zeeguu/core/reading_analysis/`, and models like `user_reading_session`, `user_browsing_session`, `user_listening_session`, and `user_word_interaction_history` indicate a feedback loop where learner behavior updates the personalization model, streaks, badges, and study recommendations.

11. **Teacher and cohort features**
   Teacher-facing functionality is first-class, not an add-on. `zeeguu/api/endpoints/teacher_dashboard/` and core models such as `teacher`, `cohort`, `teacher_cohort_map`, and `cohort_article_map` support cohort management, student overviews, article management, and student exercise/reading views. This boundary is distinct from the learner-facing API and includes permission checks plus teacher-only decorators.

12. **Notifications, onboarding, and social loop**
   User engagement features are implemented as separate subsystems: badges, daily streaks, onboarding messages, notifications, friendships, and leaderboards. These are mostly event-driven at the domain level through listeners/hooks, with persistence in dedicated tables. They reinforce the learning loop rather than serving as core content infrastructure.

13. **Batch operations and maintenance**
   The repo contains a large operational surface in `tools/` and `zeeguu/operations/`: data cleanup, backfills, migrations, reindexing, report generation, ML training, crawler jobs, and one-off fixes. This indicates the system relies heavily on offline maintenance scripts to keep the content corpus, learner model, and indexes consistent over time.

14. **Deployment and environment**
   The project is containerized (`Dockerfile`, `docker-compose.yml`, service-specific Dockerfiles, gunicorn configs) and has CI workflows for tests, Docker image publishing, and architectural diff rendering. The README shows a local setup that depends on MySQL, optional Stanza models, and environment-based configuration, so runtime boundaries are container + DB + external NLP/AI services rather than a single monolith process.

Main entry points: [`zeeguu/api/endpoints/__init__.py`](https://github.com/zeeguu/api/blob/master/zeeguu/api/endpoints/__init__.py) (Endpoints), [`zeeguu/core/model/__init__.py`](https://github.com/zeeguu/api/blob/master/zeeguu/core/model/__init__.py) (Domain models), [`zeeguu/core/content_retriever/article_downloader.py`](https://github.com/zeeguu/api/blob/master/zeeguu/core/content_retriever/article_downloader.py) (Ingest), [`zeeguu/core/llm_services/llm_service.py`](https://github.com/zeeguu/api/blob/master/zeeguu/core/llm_services/llm_service.py) (Language AI)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["API layer"]
  node_api_app["API app<br/>flask entrypoint<br/>[app.py]"]
  node_api_endpoints["Endpoints<br/>route modules<br/>[__init__.py]"]
  node_teacher_dashboard["Teacher API<br/>teacher routes<br/>[__init__.py]"]
  node_api_utils["API utils<br/>request helpers<br/>[__init__.py]"]
end

subgraph group_core["Core domain"]
  node_account_core["Accounts<br/>account domain"]
  node_content_ingest["Ingest<br/>content retrieval"]
  node_content_cleaning["Clean content<br/>content pipeline<br/>[content_cleaner.py]"]
  node_search_reco["Search & recs<br/>elastic/recommender"]
  node_nlp_core["NLP<br/>tokenization pipeline"]
  node_language_ai["Language AI<br/>LLM/translation<br/>[llm_service.py]"]
  node_audio_lessons["Audio lessons<br/>lesson pipeline"]
  node_learner_model["Learner model<br/>activity/statistics<br/>[activity.py]"]
  node_persistence[("Domain models<br/>ORM entities<br/>[__init__.py]")]
  node_social_engagement["Engagement<br/>badges/friends<br/>[friend_streak.py]"]
end

subgraph group_services["Auxiliary services"]
  node_asr_service["ASR service<br/>separate service<br/>[app.py]"]
  node_stanza_service["Stanza service<br/>separate service<br/>[app.py]"]
end

subgraph group_ops["Operations"]
  node_crawler_ops["Crawlers<br/>batch ingestion<br/>[crawl.py]"]
  node_migrations["Migrations<br/>schema evolution<br/>[README.md]"]
  node_ml_ops["ML trainers<br/>offline training"]
end

node_api_app -->|"registers"| node_api_endpoints
node_api_app -->|"uses"| node_api_utils
node_api_endpoints -->|"orchestrates"| node_account_core
node_api_endpoints -->|"orchestrates"| node_content_ingest
node_api_endpoints -->|"orchestrates"| node_search_reco
node_api_endpoints -->|"orchestrates"| node_learner_model
node_api_endpoints -->|"mounts"| node_teacher_dashboard
node_teacher_dashboard -->|"reads/writes"| node_persistence
node_content_ingest -->|"feeds"| node_content_cleaning
node_content_cleaning -->|"stores"| node_persistence
node_content_ingest -->|"stores"| node_persistence
node_content_ingest -->|"indexes"| node_search_reco
node_search_reco -->|"queries"| node_persistence
node_nlp_core -->|"enriches"| node_language_ai
node_language_ai -->|"annotates"| node_persistence
node_audio_lessons -->|"depends on"| node_language_ai
node_audio_lessons -->|"calls"| node_asr_service
node_audio_lessons -->|"calls"| node_stanza_service
node_learner_model -->|"reads/writes"| node_persistence
node_learner_model -->|"drives"| node_social_engagement
node_crawler_ops -->|"feeds"| node_content_ingest
node_migrations -.->|"evolves"| node_persistence
node_ml_ops -.->|"trains"| node_language_ai
node_nlp_core -->|"depends on"| node_stanza_service

click node_api_app "https://github.com/zeeguu/api/blob/master/zeeguu/api/app.py"
click node_api_endpoints "https://github.com/zeeguu/api/blob/master/zeeguu/api/endpoints/__init__.py"
click node_teacher_dashboard "https://github.com/zeeguu/api/blob/master/zeeguu/api/endpoints/teacher_dashboard/__init__.py"
click node_api_utils "https://github.com/zeeguu/api/blob/master/zeeguu/api/utils/__init__.py"
click node_account_core "https://github.com/zeeguu/api/blob/master/zeeguu/core/account_management/user_account_creation.py"
click node_content_ingest "https://github.com/zeeguu/api/blob/master/zeeguu/core/content_retriever/article_downloader.py"
click node_content_cleaning "https://github.com/zeeguu/api/blob/master/zeeguu/core/content_cleaning/content_cleaner.py"
click node_search_reco "https://github.com/zeeguu/api/blob/master/zeeguu/core/content_recommender/elastic_recommender.py"
click node_nlp_core "https://github.com/zeeguu/api/blob/master/zeeguu/core/tokenization/zeeguu_tokenizer.py"
click node_language_ai "https://github.com/zeeguu/api/blob/master/zeeguu/core/llm_services/llm_service.py"
click node_audio_lessons "https://github.com/zeeguu/api/blob/master/zeeguu/core/audio_lessons/daily_lesson_generator.py"
click node_learner_model "https://github.com/zeeguu/api/blob/master/zeeguu/core/user_statistics/activity.py"
click node_persistence "https://github.com/zeeguu/api/blob/master/zeeguu/core/model/__init__.py"
click node_social_engagement "https://github.com/zeeguu/api/blob/master/zeeguu/core/friends/friend_streak.py"
click node_asr_service "https://github.com/zeeguu/api/blob/master/asr_service/app.py"
click node_stanza_service "https://github.com/zeeguu/api/blob/master/stanza_service/app.py"
click node_crawler_ops "https://github.com/zeeguu/api/blob/master/zeeguu/operations/crawler/crawl.py"
click node_migrations "https://github.com/zeeguu/api/blob/master/tools/migrations/README.md"
click node_ml_ops "https://github.com/zeeguu/api/blob/master/tools/ml/cefr_trainers/train_cefr_classifiers.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api_app,node_api_endpoints,node_teacher_dashboard,node_api_utils toneBlue
class node_account_core,node_content_ingest,node_content_cleaning,node_search_reco,node_nlp_core,node_language_ai,node_audio_lessons,node_learner_model,node_persistence,node_social_engagement toneAmber
class node_asr_service,node_stanza_service toneMint
class node_crawler_ops,node_migrations,node_ml_ops toneRose
```

## Components

### API layer

HTTP app, routes, and cross-cutting request helpers

- [API app](https://github.com/zeeguu/api/blob/master/zeeguu/api/app.py): flask entrypoint · Main Zeeguu backend application bootstrap and wiring · `zeeguu/api/app.py`
- [Endpoints](https://github.com/zeeguu/api/blob/master/zeeguu/api/endpoints/__init__.py): route modules · Product API routes organized by domain · `zeeguu/api/endpoints/__init__.py`
- [Teacher API](https://github.com/zeeguu/api/blob/master/zeeguu/api/endpoints/teacher_dashboard/__init__.py): teacher routes · Teacher-only dashboard routes and permissions · `zeeguu/api/endpoints/teacher_dashboard/__init__.py`
- [API utils](https://github.com/zeeguu/api/blob/master/zeeguu/api/utils/__init__.py): request helpers · Session, security, rate limiting, background jobs, and response wrappers · `zeeguu/api/utils/__init__.py`

### Core domain

Business logic, content processing, learner model, and persistence models

- [Accounts](https://github.com/zeeguu/api/blob/master/zeeguu/core/account_management/user_account_creation.py): account domain · User creation, deletion, and account lifecycle · `zeeguu/core/account_management/user_account_creation.py`
- [Ingest](https://github.com/zeeguu/api/blob/master/zeeguu/core/content_retriever/article_downloader.py): content retrieval · Fetches articles and videos from feeds and crawlers · `zeeguu/core/content_retriever/article_downloader.py`
- [Clean content](https://github.com/zeeguu/api/blob/master/zeeguu/core/content_cleaning/content_cleaner.py): content pipeline · Normalizes text and filters low-quality or disturbing content · `zeeguu/core/content_cleaning/content_cleaner.py`
- [Search & recs](https://github.com/zeeguu/api/blob/master/zeeguu/core/content_recommender/elastic_recommender.py): elastic/recommender · Indexing, search, semantic retrieval, and personalized recommendations · `zeeguu/core/content_recommender/elastic_recommender.py`
- [NLP](https://github.com/zeeguu/api/blob/master/zeeguu/core/tokenization/zeeguu_tokenizer.py): tokenization pipeline · Tokenization, MWE detection, and linguistic pipeline support · `zeeguu/core/tokenization/zeeguu_tokenizer.py`
- [Language AI](https://github.com/zeeguu/api/blob/master/zeeguu/core/llm_services/llm_service.py): LLM/translation · LLM simplification, validation, translation, and difficulty estimation · `zeeguu/core/llm_services/llm_service.py`
- [Audio lessons](https://github.com/zeeguu/api/blob/master/zeeguu/core/audio_lessons/daily_lesson_generator.py): lesson pipeline · Lesson generation, TTS/ASR integration, and word selection · `zeeguu/core/audio_lessons/daily_lesson_generator.py`
- [Learner model](https://github.com/zeeguu/api/blob/master/zeeguu/core/user_statistics/activity.py): activity/statistics · Reading, browsing, listening, exercise tracking, streaks, and progress · `zeeguu/core/user_statistics/activity.py`
- [Domain models](https://github.com/zeeguu/api/blob/master/zeeguu/core/model/__init__.py): ORM entities · Core entities for users, content, sessions, topics, and learning state · `zeeguu/core/model/__init__.py`
- [Engagement](https://github.com/zeeguu/api/blob/master/zeeguu/core/friends/friend_streak.py): badges/friends · Badges, friendships, leaderboards, onboarding, and notifications · `zeeguu/core/friends/friend_streak.py`

### Auxiliary services

Separate runtime services for ASR and NLP model execution

- [ASR service](https://github.com/zeeguu/api/blob/master/asr_service/app.py): separate service · Standalone speech recognition runtime · `asr_service/app.py`
- [Stanza service](https://github.com/zeeguu/api/blob/master/stanza_service/app.py): separate service · Standalone NLP model runtime for tokenization and linguistic tasks · `stanza_service/app.py`

### Operations

Batch jobs, crawlers, migrations, and offline ML tasks

- [Crawlers](https://github.com/zeeguu/api/blob/master/zeeguu/operations/crawler/crawl.py): batch ingestion · Offline crawling and feed retrieval jobs · `zeeguu/operations/crawler/crawl.py`
- [Migrations](https://github.com/zeeguu/api/blob/master/tools/migrations/README.md): schema evolution · SQL-driven database evolution and backfills · `tools/migrations/README.md`
- [ML trainers](https://github.com/zeeguu/api/blob/master/tools/ml/cefr_trainers/train_cefr_classifiers.py): offline training · Batch training for CEFR and related classifiers · `tools/ml/cefr_trainers/train_cefr_classifiers.py`

## Connections

- API app → Endpoints: registers · Main app wires all API routes
- API app → API utils: uses · Shared request and response helpers
- Endpoints → Accounts: orchestrates · Account endpoints delegate to core account logic
- Endpoints → Ingest: orchestrates · Content and upload endpoints reach into ingestion
- Endpoints → Search & recs: orchestrates · Search endpoints query retrieval and recommendation
- Endpoints → Learner model: orchestrates · Sessions, stats, and activity endpoints update learner state
- Endpoints → Teacher API: mounts · Teacher dashboard is a distinct API subtree
- Teacher API → Domain models: reads/writes · Teacher and cohort views depend on core entities
- Ingest → Clean content: feeds · Raw content is cleaned before use
- Clean content → Domain models: stores · Cleaned articles and related entities are persisted
- Ingest → Domain models: stores · Downloaded content becomes domain records
- Ingest → Search & recs: indexes · Ingested content is indexed for retrieval
- Search & recs → Domain models: queries · Search and recommendation read content and learner data
- NLP → Language AI: enriches · Language processing supports translation and difficulty work
- Language AI → Domain models: annotates · AI results are stored as meanings, examples, and assessments
- Audio lessons → Language AI: depends on · Lesson generation uses LLM and translation services
- Audio lessons → ASR service: calls · ASR is provided by a separate service
- Audio lessons → Stanza service: calls · Stanza-backed NLP is externalized into a service
- Learner model → Domain models: reads/writes · Activity tracking is persisted across session and history tables
- Learner model → Engagement: drives · Behavior updates badges, streaks, friends, and notifications
- Crawlers → Ingest: feeds · Offline crawlers populate the ingestion pipeline
- Migrations → Domain models: evolves · Schema changes and backfills shape the model layer
- ML trainers → Language AI: trains · Offline ML jobs improve difficulty and classification models
- NLP → Stanza service: depends on · Some NLP features use the external stanza runtime
