# reached-hq/edu-access-intelligence architecture

> Architecture diagram of the GitHub repository reached-hq/edu-access-intelligence, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/reached-hq/edu-access-intelligence
- Repository: https://github.com/reached-hq/edu-access-intelligence

## Overview

Education Access Intelligence is an early-stage data-engineering project for evaluating education access and capacity in the Philippines. Its current workflow profiles source datasets and cross-source comparability, then ingests approved deliveries through validation into Bronze while recording batch and quality-control state. The README describes later Silver-to-dashboard layers, but those implementations are not present in the supplied tree, so they are not represented as runtime components. Edges reflect the shown excerpts, SOURCE INDEX references, or README; some ingestion internals are only partially sampled.

Main entry points: [`src/ingestion/pipeline.py`](https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/pipeline.py) (Ingestion run), [`src/ingestion/formats.py`](https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/formats.py) (Format dispatch), [`analysis/profiling`](https://github.com/reached-hq/edu-access-intelligence/tree/main/analysis/profiling) (Dataset profiling), [`src/ingestion/validate.py`](https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/validate.py) (Delivery validation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_research["Source research"]
  node_profiling_suite["Dataset profiling"]
  node_cross_source["Cross-source research"]
  node_xlsx_helper["XLSX reader<br/>[xlsx.py]"]
end

subgraph group_ingestion["Delivery ingestion"]
  node_cli["Ingestion CLI<br/>[cli.py]"]
  node_run_orchestrator["Ingestion run<br/>[pipeline.py]"]
  node_format_dispatch["Format dispatch<br/>[formats.py]"]
  node_source_contract["Source contracts<br/>[contract.py]"]
  node_batch_discovery["Batch discovery<br/>[batch.py]"]
  node_revision_handling["Revision handling<br/>[revision.py]"]
end

subgraph group_quality["Validation and loading"]
  node_archive_reader["Archive reader<br/>[archive.py]"]
  node_delivery_validation["Delivery validation<br/>[validate.py]"]
  node_workbook_validation["Workbook validation<br/>[workbook.py]"]
  node_table_validation["Table validation<br/>[xlsx_table.py]"]
  node_delivery_reader["Delivery reader<br/>[reader.py]"]
  node_bronze_loader[("Bronze loading<br/>[bronze.py]")]
end

subgraph group_operations["Run state and storage"]
  node_control_tables[("Control records<br/>[control.py]")]
  node_store_adapter[("Store adapter<br/>[store.py]")]
  node_error_model["Ingestion errors<br/>[errors.py]"]
end

node_planners(("Education planners"))
node_publishers(("Data publishers"))

node_planners -->|"frames questions"| node_cross_source
node_publishers -.->|"provide datasets"| node_profiling_suite
node_profiling_suite -.->|"compare sources"| node_cross_source
node_cross_source -->|"reads workbooks"| node_xlsx_helper
node_profiling_suite -->|"reads workbooks"| node_xlsx_helper
node_publishers -.->|"deliver files"| node_cli
node_cli -->|"starts run"| node_run_orchestrator
node_run_orchestrator -->|"selects format"| node_format_dispatch
node_run_orchestrator -->|"loads contract"| node_source_contract
node_run_orchestrator -->|"discovers batches"| node_batch_discovery
node_run_orchestrator -->|"handles revisions"| node_revision_handling
node_format_dispatch -->|"prepares archives"| node_delivery_validation
node_format_dispatch -->|"prepares sheets"| node_workbook_validation
node_format_dispatch -->|"prepares tables"| node_table_validation
node_delivery_validation -->|"checks archives"| node_archive_reader
node_workbook_validation -->|"reads delivery"| node_delivery_reader
node_table_validation -->|"reads delivery"| node_delivery_reader
node_run_orchestrator -->|"loads rows"| node_bronze_loader
node_bronze_loader -->|"writes Bronze"| node_store_adapter
node_run_orchestrator -->|"records batches"| node_control_tables
node_control_tables -->|"writes state"| node_store_adapter
node_run_orchestrator -->|"records failures"| node_error_model
node_delivery_validation -->|"rejects invalid files"| node_error_model

click node_profiling_suite "https://github.com/reached-hq/edu-access-intelligence/tree/main/analysis/profiling"
click node_cross_source "https://github.com/reached-hq/edu-access-intelligence/blob/main/analysis/cross_source/cross_source_deped.py"
click node_xlsx_helper "https://github.com/reached-hq/edu-access-intelligence/blob/main/src/profiling/xlsx.py"
click node_cli "https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/cli.py"
click node_run_orchestrator "https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/pipeline.py"
click node_format_dispatch "https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/formats.py"
click node_source_contract "https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/contract.py"
click node_batch_discovery "https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/batch.py"
click node_revision_handling "https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/revision.py"
click node_archive_reader "https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/archive.py"
click node_delivery_validation "https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/validate.py"
click node_workbook_validation "https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/workbook.py"
click node_table_validation "https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/xlsx_table.py"
click node_delivery_reader "https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/reader.py"
click node_bronze_loader "https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/bronze.py"
click node_control_tables "https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/control.py"
click node_store_adapter "https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/store.py"
click node_error_model "https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/errors.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_profiling_suite,node_cross_source,node_xlsx_helper toneBlue
class node_cli,node_run_orchestrator,node_format_dispatch,node_source_contract,node_batch_discovery,node_revision_handling toneAmber
class node_archive_reader,node_delivery_validation,node_workbook_validation,node_table_validation,node_delivery_reader,node_bronze_loader toneMint
class node_control_tables,node_store_adapter,node_error_model toneRose
class node_planners,node_publishers toneIndigo
```

## Components

### Source research

- [Dataset profiling](https://github.com/reached-hq/edu-access-intelligence/tree/main/analysis/profiling): `analysis/profiling`
- [Cross-source research](https://github.com/reached-hq/edu-access-intelligence/blob/main/analysis/cross_source/cross_source_deped.py): `analysis/cross_source/cross_source_deped.py`
- [XLSX reader](https://github.com/reached-hq/edu-access-intelligence/blob/main/src/profiling/xlsx.py): `src/profiling/xlsx.py`

### Delivery ingestion

- [Ingestion CLI](https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/cli.py): `src/ingestion/cli.py`
- [Ingestion run](https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/pipeline.py): `src/ingestion/pipeline.py`
- [Format dispatch](https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/formats.py): `src/ingestion/formats.py`
- [Source contracts](https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/contract.py): `src/ingestion/contract.py`
- [Batch discovery](https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/batch.py): `src/ingestion/batch.py`
- [Revision handling](https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/revision.py): `src/ingestion/revision.py`

### Validation and loading

- [Archive reader](https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/archive.py): `src/ingestion/archive.py`
- [Delivery validation](https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/validate.py): `src/ingestion/validate.py`
- [Workbook validation](https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/workbook.py): `src/ingestion/workbook.py`
- [Table validation](https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/xlsx_table.py): `src/ingestion/xlsx_table.py`
- [Delivery reader](https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/reader.py): `src/ingestion/reader.py`
- [Bronze loading](https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/bronze.py): `src/ingestion/bronze.py`

### Run state and storage

- [Control records](https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/control.py): `src/ingestion/control.py`
- [Store adapter](https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/store.py): `src/ingestion/store.py`
- [Ingestion errors](https://github.com/reached-hq/edu-access-intelligence/blob/main/src/ingestion/errors.py): `src/ingestion/errors.py`

### Other components

- **Education planners**
- **Data publishers**

## Connections

- Education planners → Cross-source research: frames questions (evidence: `README.md`)
- Data publishers → Dataset profiling: provide datasets
- Dataset profiling → Cross-source research: compare sources
- Cross-source research → XLSX reader: reads workbooks (evidence: `analysis/cross_source/cross_source_deped.py`)
- Dataset profiling → XLSX reader: reads workbooks (evidence: `analysis/profiling/profile_psa_poverty.py`)
- Data publishers → Ingestion CLI: deliver files
- Ingestion CLI → Ingestion run: starts run (evidence: `src/ingestion/cli.py`)
- Ingestion run → Format dispatch: selects format (evidence: `src/ingestion/pipeline.py`)
- Ingestion run → Source contracts: loads contract (evidence: `src/ingestion/pipeline.py`)
- Ingestion run → Batch discovery: discovers batches (evidence: `src/ingestion/pipeline.py`)
- Ingestion run → Revision handling: handles revisions (evidence: `src/ingestion/pipeline.py`)
- Format dispatch → Delivery validation: prepares archives (evidence: `src/ingestion/formats.py`)
- Format dispatch → Workbook validation: prepares sheets (evidence: `src/ingestion/formats.py`)
- Format dispatch → Table validation: prepares tables (evidence: `src/ingestion/formats.py`)
- Delivery validation → Archive reader: checks archives (evidence: `src/ingestion/validate.py`)
- Workbook validation → Delivery reader: reads delivery (evidence: `src/ingestion/workbook.py`)
- Table validation → Delivery reader: reads delivery (evidence: `src/ingestion/xlsx_table.py`)
- Ingestion run → Bronze loading: loads rows (evidence: `src/ingestion/pipeline.py`)
- Bronze loading → Store adapter: writes Bronze (evidence: `src/ingestion/bronze.py`)
- Ingestion run → Control records: records batches (evidence: `src/ingestion/pipeline.py`)
- Control records → Store adapter: writes state (evidence: `src/ingestion/control.py`)
- Ingestion run → Ingestion errors: records failures (evidence: `src/ingestion/pipeline.py`)
- Delivery validation → Ingestion errors: rejects invalid files (evidence: `src/ingestion/validate.py`)
