How garon-sys/mynhanes works
Project type and purpose
• MyNHANES is a full-stack Django application for managing, transforming, querying, and exporting NHANES survey data.
• It provides both a web UI (Django admin) and CLI management commands for ETL-style workflows (ingestion, transformation, export).
File structure
• Top-level:
• manage.py – Django entrypoint for server and custom commands
• mynhanes/core – Django settings, URL routing, ASGI/WGI
• mynhanes/nhanes – Main Django app containing models, migrations, admin, views, services, signals, CLI commands, and workprocess modules
• mynhanes/masterdata – CSV master data loaded into the database
• rules & mynhanes/nhanes/rules – Transformation rule definitions (Python + metadata)
• tests – Unit and fixture definitions
• Configuration:
• pyproject.toml, .pylintrc, .flake8 – project metadata and linting
• .readthedocs.yaml, docs/ – documentation build config
Key insights from the README
• Uses Python 3.12+, Django, and SQLite by default
• Core modules: Master Data Management, WorkProcess (ingestion → processing), Transformations (rule-based), Query/Export
• Entrypoints: manage.py deploy --type local to initialize DB & load master data; manage.py runserver for web UI
Main components
a. Frontend (Django Admin UI)
b. Backend Django application
– URL router → views → services → models/ORM
– Management commands → ingestion, transformation, export
– Signals for lifecycle events
c. Database (SQLite)
d. Master Data store (CSV files / initial load)
e. Rule Repository (filesystem directories with rule.py + metadata)
f. External data source (NHANES raw files downloaded during ingestion)
Relationships & interactions
• User/Admin → HTTP → Django Admin UI → Django View layer → ORM → SQLite
• CLI user → manage.py <command> → Command module → service/workprocess → ORM → SQLite
• Ingestion: downloads NHANES files → parses → writes into database tables
• Transformation: rule_manager loads Python rule modules → applies logic → updates database
• Query/Export: query builder constructs ORM queries → data retrieved → CSV output
Architectural patterns & design principles
• MVC (Django) separating models, views, templates (admin)
• ETL pipeline pattern in workprocess subpackage
• Plug-in style rule engine: each rule in its own folder, discovered and invoked at runtime
• Modular services layer (services/) to encapsulate business logic
• Use of Django signals to decouple event triggers (e.g., post-ingestion hooks)
Relevant technologies & frameworks
• Python 3.12+, Django
• SQLite (default) via Django ORM
• Sphinx for docs
• pytest/fixtures for tests
• Packaging via pip (pyproject.toml)
Essential details
• ETL stages and where rules live
• Both HTTP and CLI pathways
• With technology names (e.g., Django ORM, SQLite)
• Keep it granular: separate masterdata loader, ingestion, transformation, query, export