# sofiacper/etai-pipeline architecture

> Architecture diagram of the GitHub repository sofiacper/etai-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sofiacper/etai-pipeline
- Repository: https://github.com/sofiacper/etai-pipeline

## Overview

This repository is a configurable COMPAS recidivism prediction pipeline. Its main workflow loads the CSV, cleans rows and separates features, sets aside a locked test set, evaluates a preprocessing-plus-model pipeline with cross-validation on development data, refits on that development data, and saves reports. The reports include out-of-fold classification and race-based false-positive comparisons with COMPAS. The graph follows the sampled entry point and modules; README-documented capabilities are included where supported, while unsampled implementation details are not inferred.

Main entry points: [`main.py`](https://github.com/sofiacper/etai-pipeline/blob/main/main.py) (Orchestrator), [`src/data.py`](https://github.com/sofiacper/etai-pipeline/blob/main/src/data.py) (CSV loader), [`src/preprocessing.py`](https://github.com/sofiacper/etai-pipeline/blob/main/src/preprocessing.py) (Feature transformer), [`src/evaluate.py`](https://github.com/sofiacper/etai-pipeline/blob/main/src/evaluate.py) (Cross-validation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Data and setup"]
  node_entry["Orchestrator<br/>[main.py]"]
  node_config["Run configuration<br/>[config.yaml]"]
  node_csv["COMPAS records"]
  node_loader["CSV loader<br/>[data.py]"]
end

subgraph group_preparation["Data preparation"]
  node_cleaning["Cleaning and features<br/>[preprocessing.py]"]
  node_split["Dev/test split<br/>[preprocessing.py]"]
end

subgraph group_prediction["Modeling and evaluation"]
  node_preprocessor["Feature transformer<br/>[preprocessing.py]"]
  node_model["Model factory<br/>[model.py]"]
  node_evaluation["Cross-validation<br/>[evaluate.py]"]
  node_reports["Classification and fairness<br/>[evaluate.py]"]
end

subgraph group_reporting["Run reporting"]
  node_results["Report saver<br/>[results.py]"]
end

node_user(("Pipeline runner"))

node_user -->|"runs"| node_entry
node_entry -->|"reads"| node_config
node_entry -->|"calls"| node_loader
node_csv -->|"CSV input"| node_loader
node_loader -->|"returns rows"| node_entry
node_entry -->|"calls"| node_cleaning
node_entry -->|"calls"| node_split
node_entry -->|"builds"| node_preprocessor
node_entry -->|"builds"| node_model
node_entry -->|"cross-validates"| node_evaluation
node_entry -->|"requests reports"| node_reports
node_entry -->|"saves report"| node_results
node_evaluation -->|"OOF predictions"| node_reports
node_preprocessor -->|"pipeline stages"| node_model

click node_entry "https://github.com/sofiacper/etai-pipeline/blob/main/main.py"
click node_config "https://github.com/sofiacper/etai-pipeline/blob/main/config.yaml"
click node_csv "https://github.com/sofiacper/etai-pipeline/blob/main/data/compas_two_year_recidivism.csv"
click node_loader "https://github.com/sofiacper/etai-pipeline/blob/main/src/data.py"
click node_cleaning "https://github.com/sofiacper/etai-pipeline/blob/main/src/preprocessing.py"
click node_split "https://github.com/sofiacper/etai-pipeline/blob/main/src/preprocessing.py"
click node_preprocessor "https://github.com/sofiacper/etai-pipeline/blob/main/src/preprocessing.py"
click node_model "https://github.com/sofiacper/etai-pipeline/blob/main/src/model.py"
click node_evaluation "https://github.com/sofiacper/etai-pipeline/blob/main/src/evaluate.py"
click node_reports "https://github.com/sofiacper/etai-pipeline/blob/main/src/evaluate.py"
click node_results "https://github.com/sofiacper/etai-pipeline/blob/main/src/results.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_entry,node_config,node_csv,node_loader toneBlue
class node_cleaning,node_split toneAmber
class node_preprocessor,node_model,node_evaluation,node_reports toneMint
class node_results toneRose
class node_user toneIndigo
```

## Components

### Data and setup

- [Orchestrator](https://github.com/sofiacper/etai-pipeline/blob/main/main.py): `main.py`
- [Run configuration](https://github.com/sofiacper/etai-pipeline/blob/main/config.yaml): `config.yaml`
- [COMPAS records](https://github.com/sofiacper/etai-pipeline/blob/main/data/compas_two_year_recidivism.csv): `data/compas_two_year_recidivism.csv`
- [CSV loader](https://github.com/sofiacper/etai-pipeline/blob/main/src/data.py): `src/data.py`

### Data preparation

- [Cleaning and features](https://github.com/sofiacper/etai-pipeline/blob/main/src/preprocessing.py): `src/preprocessing.py`
- [Dev/test split](https://github.com/sofiacper/etai-pipeline/blob/main/src/preprocessing.py): `src/preprocessing.py`

### Modeling and evaluation

- [Feature transformer](https://github.com/sofiacper/etai-pipeline/blob/main/src/preprocessing.py): `src/preprocessing.py`
- [Model factory](https://github.com/sofiacper/etai-pipeline/blob/main/src/model.py): `src/model.py`
- [Cross-validation](https://github.com/sofiacper/etai-pipeline/blob/main/src/evaluate.py): `src/evaluate.py`
- [Classification and fairness](https://github.com/sofiacper/etai-pipeline/blob/main/src/evaluate.py): `src/evaluate.py`

### Run reporting

- [Report saver](https://github.com/sofiacper/etai-pipeline/blob/main/src/results.py): `src/results.py`

### Other components

- **Pipeline runner**

## Connections

- Pipeline runner → Orchestrator: runs
- Orchestrator → Run configuration: reads
- Orchestrator → CSV loader: calls
- COMPAS records → CSV loader: CSV input
- CSV loader → Orchestrator: returns rows
- Orchestrator → Cleaning and features: calls
- Orchestrator → Dev/test split: calls
- Orchestrator → Feature transformer: builds
- Orchestrator → Model factory: builds
- Orchestrator → Cross-validation: cross-validates
- Orchestrator → Classification and fairness: requests reports
- Orchestrator → Report saver: saves report
- Cross-validation → Classification and fairness: OOF predictions
- Feature transformer → Model factory: pipeline stages