# garon-sys/mynhanes architecture

> Architecture diagram of the GitHub repository garon-sys/mynhanes, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/garon-sys/mynhanes
- Repository: https://github.com/garon-sys/mynhanes

## Overview

##### Project type and purpose
- MyNHANES is a full-stack Django application for managing, transforming, querying, and exporting NHANES survey data.
- It provides both a web UI (Django admin) and CLI management commands for ETL-style workflows (ingestion, transformation, export).

##### File structure
- Top-level:
- `manage.py` – Django entrypoint for server and custom commands
- `mynhanes/core` – Django settings, URL routing, ASGI/WGI
- `mynhanes/nhanes` – Main Django app containing models, migrations, admin, views, services, signals, CLI commands, and workprocess modules
- `mynhanes/masterdata` – CSV master data loaded into the database
- `rules` & `mynhanes/nhanes/rules` – Transformation rule definitions (Python + metadata)
- `tests` – Unit and fixture definitions
- Configuration:
- `pyproject.toml`, `.pylintrc`, `.flake8` – project metadata and linting
- `.readthedocs.yaml`, `docs/` – documentation build config

##### Key insights from the README
- Uses Python 3.12+, Django, and SQLite by default
- Core modules: Master Data Management, WorkProcess (ingestion → processing), Transformations (rule-based), Query/Export
- Entrypoints: `manage.py deploy --type local` to initialize DB & load master data; `manage.py runserver` for web UI

##### Main components
a. Frontend (Django Admin UI)
b. Backend Django application
– URL router → views → services → models/ORM
– Management commands → ingestion, transformation, export
– Signals for lifecycle events
c. Database (SQLite)
d. Master Data store (CSV files / initial load)
e. Rule Repository (filesystem directories with rule.py + metadata)
f. External data source (NHANES raw files downloaded during ingestion)

##### Relationships & interactions
- User/Admin → HTTP → Django Admin UI → Django View layer → ORM → SQLite
- CLI user → `manage.py <command>` → Command module → service/workprocess → ORM → SQLite
- Ingestion: downloads NHANES files → parses → writes into database tables
- Transformation: rule_manager loads Python rule modules → applies logic → updates database
- Query/Export: query builder constructs ORM queries → data retrieved → CSV output

##### Architectural patterns & design principles
- MVC (Django) separating models, views, templates (admin)
- ETL pipeline pattern in workprocess subpackage
- Plug-in style rule engine: each rule in its own folder, discovered and invoked at runtime
- Modular services layer (`services/`) to encapsulate business logic
- Use of Django signals to decouple event triggers (e.g., post-ingestion hooks)

##### Relevant technologies & frameworks
- Python 3.12+, Django
- SQLite (default) via Django ORM
- Sphinx for docs
- pytest/fixtures for tests
- Packaging via pip (`pyproject.toml`)

##### Essential details
- ETL stages and where rules live
- Both HTTP and CLI pathways
- With technology names (e.g., Django ORM, SQLite)
- Keep it granular: separate masterdata loader, ingestion, transformation, query, export

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Presentation Layer
    subgraph "Presentation Layer"
        direction TB
        AdminUI["Django Admin UI"]:::frontend
        CLI["manage.py CLI"]:::cli
    end

    AdminUI -->|HTTP Request| Router
    CLI -->|CLI Command| CommandModule

    %% Application Layer
    subgraph "Application Layer"
        direction TB
        Router["URL Router"]:::service
        Views["Views Layer"]:::service
        Services["Services Layer"]:::service
        subgraph "WorkProcess (ETL Pipeline)"
            direction TB
            IngestWP["Ingestion WorkProcess"]:::service
            TransformWP["Transformation Manager"]:::service
            MasterdataWP["MasterData Import/Export"]:::service
            SyncWP["Sync WorkProcess"]:::service
        end
        RuleEngine["Rule Engine / Repository"]:::rule
        Reports["Reports / Query Builders"]:::service
        Signals["Signals"]:::utils
        Utils["Utilities"]:::utils
        CommandModule["Management Commands"]:::cli
    end

    Router -->|routes to| Views
    Views -->|calls| Services
    Services -->|invokes| IngestWP
    Services -->|invokes| TransformWP
    Services -->|invokes| MasterdataWP
    Services -->|invokes| SyncWP
    TransformWP -->|loads| RuleEngine
    CommandModule -->|uses| Services
    Reports -->|queries| DB
    Signals -->|hooks| Services
    Utils -->|helpers| Services

    %% Data Layer
    subgraph "Data Layer"
        direction TB
        DB["SQLite Database"]:::data
        MasterCSV["Master Data CSV"]:::data
        RawFiles["Raw NHANES Files"]:::data
        ExportCSV["Exported CSV Output"]:::data
    end

    IngestWP -->|loads from| RawFiles
    IngestWP -->|writes to| DB
    MasterdataWP -->|imports| MasterCSV
    MasterdataWP -->|writes to| DB
    TransformWP -->|updates| DB
    Reports -->|reads from| DB
    Reports -->|outputs| ExportCSV

    %% Entry Points
    AdminUI -->|entrypoint| Router
    CLI -->|entrypoint| CommandModule

    %% Click Events
    click AdminUI "https://github.com/garon-sys/mynhanes/blob/main/mynhanes/nhanes/admin.py"
    click Router "https://github.com/garon-sys/mynhanes/blob/main/mynhanes/core/urls.py"
    click Services "https://github.com/garon-sys/mynhanes/tree/main/mynhanes/nhanes/services/"
    click IngestWP "https://github.com/garon-sys/mynhanes/blob/main/mynhanes/nhanes/workprocess/ingestion_nhanes.py"
    click IngestWP "https://github.com/garon-sys/mynhanes/blob/main/mynhanes/nhanes/workprocess/ingestion_utils.py"
    click MasterdataWP "https://github.com/garon-sys/mynhanes/blob/main/mynhanes/nhanes/workprocess/masterdata_import.py"
    click MasterdataWP "https://github.com/garon-sys/mynhanes/blob/main/mynhanes/nhanes/workprocess/masterdata_export.py"
    click SyncWP "https://github.com/garon-sys/mynhanes/blob/main/mynhanes/nhanes/workprocess/sync_workprocess.py"
    click TransformWP "https://github.com/garon-sys/mynhanes/blob/main/mynhanes/nhanes/workprocess/transformation_manager.py"
    click RuleEngine "https://github.com/garon-sys/mynhanes/tree/main/mynhanes/nhanes/rules/"
    click CommandModule "https://github.com/garon-sys/mynhanes/tree/main/mynhanes/nhanes/management/commands/"
    click Reports "https://github.com/garon-sys/mynhanes/tree/main/mynhanes/nhanes/reports/"
    click Signals "https://github.com/garon-sys/mynhanes/blob/main/mynhanes/nhanes/signals.py"
    click Utils "https://github.com/garon-sys/mynhanes/tree/main/mynhanes/nhanes/utils/"
    click CLI "https://github.com/garon-sys/mynhanes/blob/main/manage.py"
    click Views "https://github.com/garon-sys/mynhanes/blob/main/mynhanes/nhanes/views.py"
    click Services "https://github.com/garon-sys/mynhanes/blob/main/mynhanes/nhanes/services/deploy.py"
    click Services "https://github.com/garon-sys/mynhanes/blob/main/mynhanes/nhanes/services/rule_manager.py"
    click Services "https://github.com/garon-sys/mynhanes/blob/main/mynhanes/nhanes/services/update_variables_tags.py"
    click Router "https://github.com/garon-sys/mynhanes/blob/main/mynhanes/core/settings.py"
    click Router "https://github.com/garon-sys/mynhanes/blob/main/mynhanes/core/asgi.py"
    click Router "https://github.com/garon-sys/mynhanes/blob/main/mynhanes/core/wsgi.py"

    classDef frontend fill:#D0E8FF,stroke:#0366D6,color:#0366D6;
    classDef cli fill:#FFE4B5,stroke:#FF8C00,color:#CC5500;
    classDef service fill:#FFF4C1,stroke:#DAA520,color:#DAA520;
    classDef rule fill:#EEDCFF,stroke:#9932CC,color:#4B0082;
    classDef data fill:#E2F0D9,stroke:#228B22,color:#006400;
    classDef utils fill:#F0F0F0,stroke:#A9A9A9,color:#696969;
```
