# tishan-jayasekera/sg-job-profitability-os architecture

> Architecture diagram of the GitHub repository tishan-jayasekera/sg-job-profitability-os, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tishan-jayasekera/sg-job-profitability-os
- Repository: https://github.com/tishan-jayasekera/sg-job-profitability-os

## Overview

##### Project Type and Purpose
- This is a full-stack, data-centric Streamlit web application (“SG Job Profitability OS”) that provides interactive analytics, forecasting, quoting, delivery control, and capacity management for job profitability.
- It combines a lightweight frontend (Streamlit pages) with a backend “data pipeline + business logic” layer that reads, processes, and aggregates Parquet data, runs metrics/models, then renders visualizations.

##### Main Architectural Components
a. Frontend (Streamlit UI)
- app.py: entry point that launches Streamlit and configures navigation.
- pages/: individual page modules (Executive Summary, Quote Builder, etc.).
- src/ui/: layout, state management, chart wrappers, formatting, and UI components.
b. Backend / Business Logic
- src/data/: loader (caching), schema validation, semantic-safe aggregations, cohort/time window logic, marts building modules.
- src/metrics/: metric calculators for profitability, capacity, rate capture, margin bridges, etc.
- src/modeling/: forecasting, intervention triggers, supply benchmarks.
- src/staffing/: staffing engine for capacity recommendations.
- src/exports.py: data/file export utilities.
c. Data Sources
- ./data/processed/: input Parquet or CSV files (fact tables, audit logs).
- ./src/data/marts/: generated mart files (active_jobs_snapshot.parquet, etc.).
- scripts/build_marts.py: offline mart-builder for pre-aggregation.
d. Configuration
- src/config.py: environment variables (DATA_DIR, APP_ENV, CACHE_TTL, recency parameters).
- .streamlit/config.toml / Dockerfile: deployment parameters.
e. Tests
- tests/: unit tests validating aggregations, reconciliation, schemas, metrics.

##### Component Interactions & Data Flow
1) Data Ingestion:
- Raw Parquet/CSV files in data/processed feed into src/data/loader (with caching).
- scripts/build_marts.py optionally pre-builds marts for performance.
2) Data Processing:
- src/data modules perform validation (schema.py), safe semantic transformations (semantic.py), cohort definitions, mart construction.
3) Business Logic:
- src/metrics fetch processed DataFrames, compute KPIs.
- src/modeling and src/staffing generate forecasts, interventions, capacity plans.
4) UI Rendering:
- Streamlit pages import from src/ui for layout, charts, tables, and invoke business logic methods.
- Results displayed as interactive dashboards, tables, drill-downs.
5) Exports & External Services:
- src/exports pushes reports or data files.
- Deployed on Streamlit Cloud or Docker; data can be pulled from cloud storage or local file system.

##### Architectural Patterns & Design Principles
- Layered Pattern: clear separation of UI / presentation, business logic (metrics, modeling), data processing, and data storage.
- Modular Design: each domain area (data, metrics, modeling, staffing, UI) is its own package.
- Caching & Performance: loader uses caching TTL; offline marts improve responsiveness.
- Single-Page Application via Streamlit: navigation through “pages” controlled by app.py.
- Configurable via environment → supports dev/prod parity and containerization.

##### Detailed Capture of Essential Elements
- Don’t omit the src/ui state management—session state is critical for reactive dashboards.
- Call out the caching layer (CACHE_TTL) that impacts reload behavior.
- Optional mart-build process as a performance optimization.
- Tests as a separate “quality gate” validating business logic and schemas.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Frontend / Presentation Layer
    subgraph "Frontend / UI" 
        direction TB
        UI_APP["app.py"]:::frontend
        UI_PAGES["pages/"]:::frontend
        UI_SRC["src/ui/"]:::frontend
    end

    %% Business Logic Layer
    subgraph "Business Logic & Processing" 
        direction TB
        CONFIG["src/config.py"]:::config
        subgraph "Data Processing" 
            direction TB
            DATA_LOADER["src/data/loader.py"]:::processing
            SCHEMA["src/data/schema.py"]:::processing
            SEMANTIC["src/data/semantic.py"]:::processing
            COHORTS["src/data/cohorts.py"]:::processing
            JOB_LC["src/data/job_lifecycle.py"]:::processing
            CLIENT_A["src/data/client_analytics.py"]:::processing
            MARTS_MODULE["src/data/marts.py"]:::processing
        end
        subgraph "Metrics" 
            direction TB
            MET_PROFIT["src/metrics/profitability.py"]:::processing
            MET_CAP["src/metrics/capacity.py"]:::processing
            MET_RATE_C["src/metrics/rate_capture.py"]:::processing
            MET_RATE_B["src/metrics/rate_bridge.py"]:::processing
            MET_MARGIN["src/metrics/margin_bridge.py"]:::processing
            MET_DELIV["src/metrics/delivery_control.py"]:::processing
            MET_REV["src/metrics/revenue_reconciliation.py"]:::processing
            MET_JOBMIX["src/metrics/job_mix.py"]:::processing
            MET_SUB["src/metrics/client_group_subsidy.py"]:::processing
            MET_TIME["src/metrics/time_allocation.py"]:::processing
            MET_ACTIVE["src/metrics/active_projects.py"]:::processing
            MET_QUOTE["src/metrics/quote_delivery.py"]:::processing
            MET_SEG["src/metrics/segment_profiling.py"]:::processing
        end
        subgraph "Modeling & Staffing" 
            direction TB
            MOD_FORE["src/modeling/forecast.py"]:::processing
            MOD_BENCH["src/modeling/benchmarks.py"]:::processing
            MOD_INTERV["src/modeling/intervention.py"]:::processing
            MOD_SUPPLY["src/modeling/supply.py"]:::processing
            STAFF_ENGINE["src/staffing/engine.py"]:::processing
        end
        EXPORTS["src/exports.py"]:::processing
    end

    %% Data Storage Layer
    subgraph "Data Storage" 
        direction TB
        PROCESSED["src/data/processed/"]:::storage
        MARTS["src/data/marts/"]:::storage
    end

    %% Scripts & Tools
    subgraph "Scripts & Tools" 
        direction TB
        BUILD_MARTS["scripts/build_marts.py"]:::scripts
        REFRESH["scripts/refresh_processed_data.sh"]:::scripts
        VALIDATE["scripts/validate_inputs.py"]:::scripts
    end

    %% Tests
    TESTS["tests/"]:::tests

    %% Data Flows
    PROCESSED -->|Parquet files| DATA_LOADER
    BUILD_MARTS -->|pre-aggregation| MARTS
    DATA_LOADER --> SCHEMA --> SEMANTIC --> COHORTS --> JOB_LC --> CLIENT_A --> MARTS_MODULE
    MARTS --> MARTS_MODULE
    MARTS_MODULE -->|DataFrame| MET_PROFIT
    MARTS_MODULE --> MET_CAP
    MARTS_MODULE --> MET_RATE_C
    MARTS_MODULE --> MET_RATE_B
    MARTS_MODULE --> MET_MARGIN
    MARTS_MODULE --> MET_DELIV
    MARTS_MODULE --> MET_REV
    MARTS_MODULE --> MET_JOBMIX
    MARTS_MODULE --> MET_SUB
    MARTS_MODULE --> MET_TIME
    MARTS_MODULE --> MET_ACTIVE
    MARTS_MODULE --> MET_QUOTE
    MARTS_MODULE --> MET_SEG

    MET_PROFIT --> MOD_FORE
    MET_CAP --> STAFF_ENGINE
    MOD_FORE --> MOD_INTERV --> MOD_BENCH --> MOD_SUPPLY

    MET_PROFIT -->|KPIs| UI_APP
    MET_CAP --> UI_APP
    MOD_FORE --> UI_APP
    STAFF_ENGINE --> UI_APP

    UI_APP --> UI_PAGES --> UI_SRC
    UI_SRC --> UI_PAGES

    EXPORTS --> PROCESSED
    EXPORTS --> MARTS

    CONFIG -.-> DATA_LOADER
    CONFIG -.-> UI_APP

    TESTS -.-> DATA_LOADER
    TESTS -.-> SCHEMA
    TESTS -.-> MET_PROFIT

    %% Click Events
    click UI_APP "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/app.py"
    click UI_PAGES "https://github.com/tishan-jayasekera/sg-job-profitability-os/tree/main/pages/"
    click UI_SRC "https://github.com/tishan-jayasekera/sg-job-profitability-os/tree/main/src/ui/"
    click DATA_LOADER "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/src/data/loader.py"
    click SCHEMA "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/src/data/schema.py"
    click SEMANTIC "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/src/data/semantic.py"
    click COHORTS "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/src/data/cohorts.py"
    click JOB_LC "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/src/data/job_lifecycle.py"
    click CLIENT_A "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/src/data/client_analytics.py"
    click MARTS_MODULE "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/src/data/marts.py"
    click BUILD_MARTS "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/scripts/build_marts.py"
    click MARTS "https://github.com/tishan-jayasekera/sg-job-profitability-os/tree/main/src/data/marts/"
    click PROCESSED "https://github.com/tishan-jayasekera/sg-job-profitability-os/tree/main/src/data/processed/"
    click MET_PROFIT "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/src/metrics/profitability.py"
    click MET_CAP "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/src/metrics/capacity.py"
    click MET_RATE_C "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/src/metrics/rate_capture.py"
    click MET_RATE_B "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/src/metrics/rate_bridge.py"
    click MET_MARGIN "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/src/metrics/margin_bridge.py"
    click MET_DELIV "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/src/metrics/delivery_control.py"
    click MET_REV "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/src/metrics/revenue_reconciliation.py"
    click MET_JOBMIX "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/src/metrics/job_mix.py"
    click MET_SUB "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/src/metrics/client_group_subsidy.py"
    click MET_TIME "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/src/metrics/time_allocation.py"
    click MET_ACTIVE "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/src/metrics/active_projects.py"
    click MET_QUOTE "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/src/metrics/quote_delivery.py"
    click MET_SEG "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/src/metrics/segment_profiling.py"
    click MOD_FORE "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/src/modeling/forecast.py"
    click MOD_BENCH "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/src/modeling/benchmarks.py"
    click MOD_INTERV "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/src/modeling/intervention.py"
    click MOD_SUPPLY "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/src/modeling/supply.py"
    click STAFF_ENGINE "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/src/staffing/engine.py"
    click EXPORTS "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/src/exports.py"
    click CONFIG "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/src/config.py"
    click REFRESH "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/scripts/refresh_processed_data.sh"
    click VALIDATE "https://github.com/tishan-jayasekera/sg-job-profitability-os/blob/main/scripts/validate_inputs.py"
    click TESTS "https://github.com/tishan-jayasekera/sg-job-profitability-os/tree/main/tests/"

    %% Styles
    classDef frontend fill:#FFA500,stroke:#333,stroke-width:1px
    classDef processing fill:#90EE90,stroke:#333,stroke-width:1px
    classDef storage fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef scripts fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef tests fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef config fill:#D3D3D3,stroke:#333,stroke-width:1px
```
