How katherinewyers/tool-dq-workbench works
Purpose and operating model
Data Quality Workbench monitors DHIS2 data quality over time: validation-rule violations, outliers, and metadata-integrity issues. It writes resulting metrics back to DHIS2 as ordinary data values, enabling DHIS2-native charts, maps, and pivots. Configuration is created interactively, then executed separately on a schedule.
Entry points and deployment
• CLI monitoring entry point: dq_monitor.py and app/cli.py; intended for scheduled dq-monitor --config ... runs.
• Local configuration UI: app/web/app.py, with routes registered from app/web/routes/ui_blueprint.py and app/web/routes/api_blueprint.py.
• Containerized execution is defined by Dockerfile and docker-compose.yml.
• Windows packaged startup is represented by installer/main_win.py and installer specifications under installer/.
Configuration and connection boundary
Configuration loading and normalization are centralized in app/core/config_loader.py; the initial/default structure is exemplified by config/sample_config.yml. The web UI edits server credentials and analysis stages, including app/web/routes/edit_server.py and the per-stage edit routes. DHIS2 base URL and API token may come from environment variables or persisted configuration. The application has no built-in UI authentication, so deployment must remain local or trusted.
Analysis pipeline
The CLI reads configured stages, obtains DHIS2 data via shared API handling in app/core/api_utils.py, then dispatches the applicable analyzer:
• Validation-rule tracking: app/analyzers/rule_analyzer.py
• Outlier tracking: app/analyzers/outlier_analyzer.py
• Metadata integrity tracking: app/analyzers/integrity_analyzer.py
• Stage orchestration/representation: app/analyzers/stage_analyzer.py
Each stage converts DHIS2 quality findings into snapshot values suitable for posting to configured DHIS2 destination data elements.
DHIS2 writeback and temporal model
DHIS2 is the material external service and both source of quality findings and destination for generated metrics. Period and numeric semantics are isolated in app/core/period_type.py, app/core/period_utils.py, and app/core/numeric_value_types.py; these form the boundary that maps collected snapshots to DHIS2-compatible reporting periods and values. UID utilities in app/core/uid_utils.py support DHIS2 identifier handling.
Min/max generation subsystem
A distinct min/max workflow calculates thresholds/statistics rather than directly running a monitoring stage. Its domain model and aggregation logic are in app/minmax/min_max_factory.py, app/minmax/min_max_statistics.py, and app/minmax/min_max_results_tracker.py. The UI exposes separate min/max pages and execution endpoints through app/web/routes/index_minmax.py, app/web/routes/minmax_analysis.py, and app/web/routes/run_minmax_stage.py.
Web interaction boundary
The Flask-style web layer renders server and stage forms, invokes configuration mutations, and can execute stages for interactive feedback. Core run paths are app/web/routes/run.py and app/web/routes/run_stage.py; stage deletion/editing is route-specific. Templates under app/web/templates/ and static Bootstrap/JavaScript assets provide a server-rendered local UI rather than a separate frontend runtime.
Main entry points: api_utils.py DHIS2 API handling · stage_analyzer.py Stage orchestration · config_loader.py Configuration loader · app.py Web application
Components 17
Entry points
- Monitor CLI · Scheduled dq-monitor execution.
dq_monitor.py - Container deployment
Dockerfile
Local web UI
Server-rendered configuration and interactive execution UI.
- Web application · Local trusted configuration UI.
app/web/app.py - UI routes
app/web/routes/ui_blueprint.py - API routes
app/web/routes/api_blueprint.py - Interactive stage runs
app/web/routes/run_stage.py
Core execution
- Configuration loader · Loads and normalizes persisted and environment-supplied settings.
app/core/config_loader.py - Sample configuration
config/sample_config.yml - DHIS2 API handling
app/core/api_utils.py - Period and value mapping · Maps snapshots to DHIS2 reporting periods and numeric values.
app/core/period_type.py
Quality analysis
- Stage orchestration · Selects and represents configured monitoring stages.
app/analyzers/stage_analyzer.py - Validation-rule tracker
app/analyzers/rule_analyzer.py - Outlier tracker
app/analyzers/outlier_analyzer.py - Metadata-integrity tracker
app/analyzers/integrity_analyzer.py
Min/max workflow
- Min/max UI and execution · Separate threshold-generation workflow.
app/web/routes/run_minmax_stage.py - Min/max statistics · Calculates threshold statistics rather than monitoring snapshots.
app/minmax/min_max_factory.py
Other components
- DHIS2 · Supplies quality findings and stores generated metric data values.
Connections 23
- Monitor CLI → Configuration loader: loads config
- Monitor CLI → Stage orchestration: runs stages
- Web application → UI routes: registers
- Web application → API routes: registers
- Web application → Interactive stage runs: serves
- UI routes → Configuration loader: edits settings
- API routes → Configuration loader: mutates config
- Interactive stage runs → Stage orchestration: executes
- Sample configuration → Configuration loader: initial structure
- Configuration loader → DHIS2 API handling: provides credentials
- Stage orchestration → Validation-rule tracker: dispatches
- Stage orchestration → Outlier tracker: dispatches
- Stage orchestration → Metadata-integrity tracker: dispatches
- Validation-rule tracker → DHIS2 API handling: queries findings
- Outlier tracker → DHIS2 API handling: queries findings
- Metadata-integrity tracker → DHIS2 API handling: queries findings
- Stage orchestration → Period and value mapping: maps snapshots
- Period and value mapping → DHIS2 API handling: formats writeback values
- DHIS2 API handling → DHIS2: reads findings / posts metrics
- Min/max UI and execution → Min/max statistics: runs
- Min/max statistics → DHIS2 API handling: retrieves data
- Container deployment → Monitor CLI: runs scheduled command
- Container deployment → Web application: hosts locally