# phil19851/test11 architecture

> Architecture diagram of the GitHub repository phil19851/test11, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/phil19851/test11
- Repository: https://github.com/phil19851/test11

## Overview

##### Project Type and Purpose
- This is a batch-oriented data-pipeline and machine-learning application written in Python.
- Purpose: extract claim data from SQL Server, enrich it via feature engineering and Azure OpenAI embeddings, train a classifier (RandomForest/XGBoost), evaluate performance, export results, and send reports via email.

a. Data Sources
- SQL Server (Prepay, Postpay, HCDC tables)
- (Optionally: any file-based inputs or config files)
b. Data Extraction & Merging
- Python modules/scripts using pyodbc for each claim type
- Merging logic for Prepay/Postpay/HCDC
c. Feature Engineering & Enrichment
- Diagnosis & Procedure Embeddings (Azure OpenAI Embedding API)
- Provider Agreement Details lookup (possibly another table in SQL Server)
- Line-level metrics calculations
- Agreement-level joins
d. Model Training & Evaluation
- Scikit-learn / XGBoost training scripts
- Evaluation module producing AUC, decile tables, precision-recall
e. Output Generation
- Excel exports (`train_Df.xlsx`, `dep.xlsx`, `Top_Feature_Importances.xlsx`, `Decile_Table_RF1.xlsx`)
- Model pickle (`random_forest_error_cat_best.pkl`)
f. Reporting
- Email automation via Python’s smtplib sending summary reports and attachments
g. Dashboard Integration (optional)
- Data published or saved for downstream dashboard/reporting systems

##### External Services & Dependencies
- Azure OpenAI Embeddings API (external REST service)
- SQL Server (via pyodbc)
- SMTP server (via smtplib)
- File system (for Excel/pickle outputs)

- Extraction Modules → Merging Module
- Merged Data → Feature Engineering Module
- Feature Engineering → Model Training & Evaluation
- Model & Metrics → Output Generation
- Outputs → Email Reporting
- Optionally: Outputs → Dashboard

##### Architectural Patterns & Principles
- Layered Batch Pipeline: clearly separate Extraction, Transformation (enrichment), Model, and Reporting layers.
- Modular code organization: one Python module per logical step (e.g., `extract.py`, `enrich.py`, `train.py`, `report.py`).
- Single responsibility principle: each module does one thing (extraction, enrichment, training, etc.).
- Externalize configuration: database credentials and API keys loaded from environment or config file.

##### Final Checklist
- [ ] Clear, descriptive labels on every component.
- [ ] Include notes on config management (e.g., environment variables for DB and Azure keys).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Documentation
    doc["README.md"]:::external
    click doc "https://github.com/phil19851/test11/blob/main/README.md"

    %% Data Layer
    subgraph "Data Layer"
        SQL[(SQL Server)]:::data
    end

    %% Pipeline Modules Directory
    pipeline["Pipeline Modules"]:::processing
    click pipeline "https://github.com/phil19851/test11/tree/main/test11/"

    %% Processing Layer
    subgraph "Processing Layer"
        ext["extract.py\n(Merge: Prepay/Postpay/HCDC)"]:::processing
        enrich["enrich.py\n(Embeddings, Provider Lookup, Line Metrics)"]:::processing
    end

    %% Model Layer
    subgraph "Model Layer"
        train["train.py\n(RandomForest/XGBoost)"]:::model
        eval["evaluate.py\n(Performance Metrics)"]:::model
    end

    %% Output & Reporting Layer
    subgraph "Output & Reporting Layer"
        output["output.py\n(Excel & Pickle Writes)"]:::report
        report["report.py\n(Email via SMTP)"]:::report
    end

    %% External Services
    subgraph "External Services"
        AZ["Azure OpenAI Embedding API"]:::external
        SMTP["SMTP Server"]:::external
    end

    %% Data Flow
    doc -->|"Configuration" | SQL
    SQL -->|"Raw Claim Rows"| ext
    ext -->|"Merged DataFrame"| enrich
    enrich -->|"Feature-enriched DataFrame"| train
    train -->|"Trained Model"| eval
    eval -->|"Model + Metrics"| output
    output -->|"Reports & Attachments"| report

    enrich -->|"Embedding Requests"| AZ
    report -->|"Send Email"| SMTP

    %% Group pipeline modules under directory
    pipeline --> ext
    pipeline --> enrich
    pipeline --> train
    pipeline --> eval
    pipeline --> output
    pipeline --> report

    %% Styles
    classDef data fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef processing fill:#90EE90,stroke:#000,stroke-width:1px
    classDef model fill:#FFA500,stroke:#000,stroke-width:1px
    classDef report fill:#DDA0DD,stroke:#000,stroke-width:1px
    classDef external fill:#f2f2f2,stroke:#555,stroke-width:1px,stroke-dasharray: 5 5
```
