How orionshinesbright/global-gdp-analysis works
This is a configurable, concurrent dataset analysis and visualization pipeline. The main workflow streams CSV/JSON rows through field mapping and integrity checks, computes a sliding average, then emits console output and live charts in a browser dashboard. Separate branches monitor queue backpressure and expose pipeline status and latest-packet data. The sampled entry point shows process wiring and queue flow; unsampled output modules are included because the source index and README identify them, but uncertain internal links are omitted.
Main entry points: main.py Bootstrap · pipeline_telemetry.py Queue telemetry · protocols.py Output manager · Stream.py Raw queue
Components 19
Pipeline orchestration
- Bootstrap
main.py - Pipeline config
config.json - Raw queue
stream/Stream.py - Intermediate queue
stream/Stream.py - Processed queue
stream/Stream.py
Data processing
- Dataset reader
plugins/input/data_loader.py - Transformer
core/data_processor.py - Stream contracts
core/protocols.py - Sliding aggregator
core/aggregator.py
Output and visualization
- Output manager
plugins/output/protocols.py - Console writer
plugins/output/console_writer.py - Chart writer
plugins/output/chart_writer.py - Chart renderer
plugins/output/chart_implementations.py - Web dashboard server
plugins/output/web/server.py - Dashboard template
plugins/output/web/templates/index.html
Pipeline monitoring
- Queue monitor
plugins/output/pipeline_monitor.py - Queue telemetry
plugins/output/pipeline_telemetry.py
Other components
- Operator
- Browser user
Connections 27
- Operator → Bootstrap: starts
README.md - Bootstrap → Pipeline config: loads
main.py - Bootstrap → Dataset reader: starts reader
main.py - Bootstrap → Transformer: starts workers
main.py - Bootstrap → Sliding aggregator: starts
main.py - Bootstrap → Output manager: starts
main.py - Bootstrap → Queue monitor: configures
main.py - Dataset reader → Raw queue: writes rows
plugins/input/data_loader.py - Transformer → Raw queue: reads rows
core/data_processor.py - Transformer → Intermediate queue: writes transformed
core/data_processor.py - Sliding aggregator → Intermediate queue: reads packets
core/aggregator.py - Sliding aggregator → Processed queue: writes averages
core/aggregator.py - Output manager → Processed queue: reads packets
plugins/output/protocols.py - Output manager → Console writer: writes output
plugins/output/protocols.py - Output manager → Chart writer: renders charts
plugins/output/protocols.py - Chart writer → Chart renderer: updates plots
plugins/output/chart_writer.py - Chart renderer → Web dashboard server: starts server
plugins/output/chart_implementations.py - Web dashboard server → Dashboard template: renders page
plugins/output/web/server.py - Browser user → Web dashboard server: views dashboard
- Bootstrap → Raw queue: creates queue
main.py - Bootstrap → Intermediate queue: creates queue
main.py - Bootstrap → Processed queue: creates queue
main.py - Queue monitor → Queue telemetry: starts polling
plugins/output/pipeline_monitor.py - Queue telemetry → Queue monitor: notifies snapshots
plugins/output/pipeline_telemetry.py - Queue telemetry → Raw queue: polls size
plugins/output/pipeline_telemetry.py - Queue telemetry → Intermediate queue: polls size
plugins/output/pipeline_telemetry.py - Queue telemetry → Processed queue: polls size
plugins/output/pipeline_telemetry.py