# martinkimoe/api architecture

> Architecture diagram of the GitHub repository martinkimoe/api, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/martinkimoe/api
- Repository: https://github.com/martinkimoe/api

## Overview

##### Project Type and Purpose
- This is a Python-based, monolithic backend API (Zeeguu-API) that exposes REST endpoints to track and model language learning.
- It is deployed as the backend for zeeguu.org and integrates with mobile/web clients, teacher dashboards, and batch tooling for feed crawling, indexing, and migrations.

##### High-Level File Structure Analysis
- zeeguu/api: HTTP routing layer (Flask-style endpoints served via WSGI/Apache2).
- zeeguu/core: Domain/business logic modules (account management, content retrieval, recommendation, NLP pipeline, emailer, scheduling, semantic search, etc.).
- zeeguu/core/model & zeeguu/core/sql: ORM + raw SQL models & queries (MySQL).
- tools: CLI utilities, data migrations, feed crawlers, batch scripts, report generators.
- config & docker files: Docker/Docker-compose setup, environment/config templates.

##### Main Components
A. API Layer
– “zeeguu/api/app.py” & “cross_domain_app.py” (entry points)
– Endpoint modules under zeeguu/api/endpoints (accounts, articles, sessions, teacher_dashboard, etc.)
– Utilities & middleware (authentication, CORS, JSON serialization)

B. Business-Logic Layer
– Core modules under zeeguu/core (content_retriever, content_cleaning, recommender, behavioral_modeling, nlp_pipeline, emailer, etc.)
– ML models & artifacts under zeeguu/core/ml_models

C. Persistence Layer
– MySQL relational database (zeeguu core/model)
– ElasticSearch index for semantic & text search (zeeguu/core/elastic)

D. External Services & Libraries
– Third-party NLP: Stanza, spaCy (zeeguu/core/tokenization, nlp_pipeline)
– Readability server & newspaper3k for parsing articles
– External translation services (if configured)
– Cron/batch runner invoking “tools/*” scripts

E. Infrastructure & DevOps
– Docker containers (API, MySQL, ElasticSearch) orchestrated via docker-compose
– Apache2 with mod_wsgi or gunicorn for serving the WSGI app
– CI pipelines (GitHub Actions under .github/workflows)
– Pre-commit hooks (in .githooks rules)

##### Relationships & Data Flows
- Client requests (web/mobile) → API Layer → Business Logic → Persistence (MySQL / ElasticSearch) → API Layer → Client
- Feed crawler (tools/article_crawler, tools/download_feed) → Business Logic → Persistence
- NLP pipeline modules ingest raw text from DB → annotate/clean → update DB / feed ElasticSearch
- Recommender modules pull user history from MySQL → compute suggestions → store feedback
- Emailer modules read events from business logic → deliver notifications (SMTP)
- Migrations scripts (tools/migrations) → schema changes in MySQL

##### Architectural Patterns & Principles
- Layered Architecture (Presentation / API, Domain Logic, Persistence)
- Hexagonal/Ports & Adapters (the core modules are independent of HTTP or DB adapters)
- Modular design: each `zeeguu/core/<feature>` is a self-contained domain service
- Separation of concerns: CLI tools vs. long-running API service

##### Technologies & Frameworks
- Python 3.x, Flask-style routing, WSGI (Apache2/mod_wsgi or gunicorn)
- SQLAlchemy or similar ORM + raw SQL
- MySQL 5.7 (relational store)
- ElasticSearch (semantic & text search index)
- Docker + Docker-compose for local development & deployment
- NLP: Stanza, spaCy, custom pipelines in zeeguu/core/nlp_pipeline
- Joblib models for paywall detection, custom ML in zeeguu/core/ml_models

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Clients
    client[/"Web & Mobile Clients"/]:::external

    %% API Layer
    subgraph "API Layer"
        direction TB
        app["app.py"]:::service
        cross["cross_domain_app.py"]:::service
        endpoints["API Endpoints"]:::service
        utils["API Utilities & Middleware"]:::service
    end

    %% Business Logic Layer
    subgraph "Business Logic"
        direction TB
        content_retriever["Content Retriever"]:::service
        content_cleaning["Content Cleaning"]:::service
        recommender["Content Recommendation"]:::service
        behavioral["Behavioral Modeling"]:::service
        nlp["NLP Pipeline"]:::service
        emailer["Emailer & Notifications"]:::service
        scheduling["Word Scheduling"]:::service
        ml_models["ML Models & Artifacts"]:::service
        tokenization["Tokenization Services"]:::service
    end

    %% Persistence Layer
    subgraph "Persistence Layer"
        direction TB
        mysql["MySQL (ORM Models)"]:::db
        rawsql["Raw SQL Queries"]:::db
        es["ElasticSearch Index"]:::db
    end

    %% External Services
    subgraph "External Services"
        direction TB
        nlp3["3rd-party NLP (Stanza, spaCy)"]:::external
        readability["Readability Server & newspaper3k"]:::external
        translation["External Translation API"]:::external
        smtp["SMTP Service"]:::external
    end

    %% Batch & CLI Tools
    subgraph "Batch & CLI Tools"
        direction TB
        tools["Tools & Scripts"]:::batch
    end

    %% Infrastructure & DevOps
    subgraph "Infrastructure & DevOps"
        direction TB
        docker["Docker & Compose"]:::infra
        ci["CI/CD Pipelines"]:::infra
        hooks["Pre-commit Hooks"]:::infra
        config_loader["Config Loader"]:::infra
    end

    %% Data Flows
    client --> app
    app --> endpoints
    endpoints --> utils
    endpoints --> content_retriever
    endpoints --> content_cleaning
    endpoints --> recommender
    endpoints --> behavioral
    endpoints --> nlp
    endpoints --> emailer
    endpoints --> scheduling
    endpoints --> ml_models
    endpoints --> tokenization

    content_retriever --> mysql
    content_cleaner --> mysql
    recommender --> es
    behavioral --> mysql
    nlp --> mysql
    nlp --> es
    emailer --> smtp
    scheduling --> mysql
    ml_models --> behavioural

    tools --> content_retriever
    tools --> mysql
    tools --> rawsql

    nlp --> nlp3
    content_retriever --> readability
    endpoints --> translation

    %% Infrastructure connections
    app --> docker
    mysql --> docker
    es --> docker
    docker --> ci
    hooks --> ci
    config_loader --> app
    config_loader --> docker

    %% Click Events
    click app "https://github.com/martinkimoe/api/blob/master/zeeguu/api/app.py"
    click cross "https://github.com/martinkimoe/api/blob/master/zeeguu/api/cross_domain_app.py"
    click endpoints "https://github.com/martinkimoe/api/tree/master/zeeguu/api/endpoints/"
    click utils "https://github.com/martinkimoe/api/tree/master/zeeguu/api/utils/"
    click content_retriever "https://github.com/martinkimoe/api/tree/master/zeeguu/core/content_retriever/"
    click content_cleaning "https://github.com/martinkimoe/api/tree/master/zeeguu/core/content_cleaning/"
    click recommender "https://github.com/martinkimoe/api/tree/master/zeeguu/core/content_recommender/"
    click behavioral "https://github.com/martinkimoe/api/tree/master/zeeguu/core/behavioral_modeling/"
    click nlp "https://github.com/martinkimoe/api/tree/master/zeeguu/core/nlp_pipeline/"
    click emailer "https://github.com/martinkimoe/api/tree/master/zeeguu/core/emailer/"
    click scheduling "https://github.com/martinkimoe/api/tree/master/zeeguu/core/word_scheduling/"
    click ml_models "https://github.com/martinkimoe/api/tree/master/zeeguu/core/ml_models/"
    click tokenization "https://github.com/martinkimoe/api/tree/master/zeeguu/core/tokenization/"
    click mysql "https://github.com/martinkimoe/api/tree/master/zeeguu/core/model/"
    click rawsql "https://github.com/martinkimoe/api/tree/master/zeeguu/core/sql/"
    click es "https://github.com/martinkimoe/api/tree/master/zeeguu/core/elastic/"
    click tools "https://github.com/martinkimoe/api/tree/master/tools/"
    click docker "https://github.com/martinkimoe/api/blob/master/docker-compose.yml"
    click ci "https://github.com/martinkimoe/api/tree/master/.github/workflows/"
    click hooks "https://github.com/martinkimoe/api/tree/master/.githooks/"
    click config_loader "https://github.com/martinkimoe/api/blob/master/zeeguu/config/loader.py"

    %% Styles
    classDef service fill:#cce5ff,stroke:#000;
    classDef db fill:#ffebcc,stroke:#000;
    classDef external fill:#e2e3e5,stroke:#000;
    classDef batch fill:#d1e7dd,stroke:#000;
    classDef infra fill:#f8d7da,stroke:#000;
    classDef externalClient fill:#cfe2ff,stroke:#000;
    class client externalClient
```
