How tishan-jayasekera/sg-job-profitability-os works
Project Type and Purpose
• This is a full-stack, data-centric Streamlit web application (“SG Job Profitability OS”) that provides interactive analytics, forecasting, quoting, delivery control, and capacity management for job profitability.
• It combines a lightweight frontend (Streamlit pages) with a backend “data pipeline + business logic” layer that reads, processes, and aggregates Parquet data, runs metrics/models, then renders visualizations.
Main Architectural Components
a. Frontend (Streamlit UI)
• app.py: entry point that launches Streamlit and configures navigation.
• pages/: individual page modules (Executive Summary, Quote Builder, etc.).
• src/ui/: layout, state management, chart wrappers, formatting, and UI components.
b. Backend / Business Logic
• src/data/: loader (caching), schema validation, semantic-safe aggregations, cohort/time window logic, marts building modules.
• src/metrics/: metric calculators for profitability, capacity, rate capture, margin bridges, etc.
• src/modeling/: forecasting, intervention triggers, supply benchmarks.
• src/staffing/: staffing engine for capacity recommendations.
• src/exports.py: data/file export utilities.
c. Data Sources
• ./data/processed/: input Parquet or CSV files (fact tables, audit logs).
• ./src/data/marts/: generated mart files (active_jobs_snapshot.parquet, etc.).
• scripts/build_marts.py: offline mart-builder for pre-aggregation.
d. Configuration
• src/config.py: environment variables (DATA_DIR, APP_ENV, CACHE_TTL, recency parameters).
• .streamlit/config.toml / Dockerfile: deployment parameters.
e. Tests
• tests/: unit tests validating aggregations, reconciliation, schemas, metrics.
Component Interactions & Data Flow
1) Data Ingestion:
• Raw Parquet/CSV files in data/processed feed into src/data/loader (with caching).
• scripts/build_marts.py optionally pre-builds marts for performance.
2) Data Processing:
• src/data modules perform validation (schema.py), safe semantic transformations (semantic.py), cohort definitions, mart construction.
3) Business Logic:
• src/metrics fetch processed DataFrames, compute KPIs.
• src/modeling and src/staffing generate forecasts, interventions, capacity plans.
4) UI Rendering:
• Streamlit pages import from src/ui for layout, charts, tables, and invoke business logic methods.
• Results displayed as interactive dashboards, tables, drill-downs.
5) Exports & External Services:
• src/exports pushes reports or data files.
• Deployed on Streamlit Cloud or Docker; data can be pulled from cloud storage or local file system.
Architectural Patterns & Design Principles
• Layered Pattern: clear separation of UI / presentation, business logic (metrics, modeling), data processing, and data storage.
• Modular Design: each domain area (data, metrics, modeling, staffing, UI) is its own package.
• Caching & Performance: loader uses caching TTL; offline marts improve responsiveness.
• Single-Page Application via Streamlit: navigation through “pages” controlled by app.py.
• Configurable via environment → supports dev/prod parity and containerization.
Detailed Capture of Essential Elements
• Don’t omit the src/ui state management—session state is critical for reactive dashboards.
• Call out the caching layer (CACHE_TTL) that impacts reload behavior.
• Optional mart-build process as a performance optimization.
• Tests as a separate “quality gate” validating business logic and schemas.