# it-gssb/sycamore architecture

> Architecture diagram of the GitHub repository it-gssb/sycamore, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/it-gssb/sycamore
- Repository: https://github.com/it-gssb/sycamore

## Overview

##### Project type and purpose
- This is a command-line Python ETL tool whose primary purpose is to extract student roster data from the Sycamore API, cache it locally, transform it into various target formats (SDS 2.1, Clever import, etc.), and output CSVs. It also ships a set of PowerShell scripts for account-management tasks against an Active Directory (or similar) after the CSVs are generated.

##### File structure
- src/lib: low-level modules (SycamoreRest, caching, entity models, generators) that encapsulate the HTTP client, local disk cache, and in-memory representations of Sycamore data.
- src/extract: pipeline stages that read from the cache or API, validate and filter, project CSV file schemas, and merge multiple CSV fragments.
- src/sds: “consumer” scripts that orchestrate the end-to-end flow for SDS 2.1 and SDS-to-Clever data generation. These scripts invoke extract modules and write “outv21” or “outClever” directories.
- src/clever: transformer module for Clever-specific data shaping.
- src/powershell: standalone PS1 scripts for creating/deleting groups, setting passwords, toggling account enabled status, using the CSVs to drive AD.

##### Key insights from the README
- The workflow is: CLI invocation → check/cache Sycamore data → transformation pipelines → write CSV output.
- External dependencies: Sycamore API (requires access token), Python libs (pandas, phonenumbers).
- Two major “consumers”: SDS 2.1 importer and Clever importer.

##### Capturing essential details
- That both SDS and Clever exporters depend on the same upstream extract modules.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Systems
    SYCAPI["Sycamore API"]:::external
    AD["Active Directory"]:::external

    %% CLI Layer
    subgraph "CLI Layer"
        SDS1["ExtractFromSycamoreToSDS21.py"]:::cli
        SDS2["ExtractFromSycamoreToSDSClever.py"]:::cli
        SDS3["ExtractStudents.py"]:::cli
        CLEVER["CleverTransformer.py"]:::cli
    end

    %% Core Library
    subgraph "Core Library"
        REST["SycamoreRest Client"]:::core
        CACHEM["Local Disk Cache Manager"]:::core
        ENT["Entity Models"]:::core
        GEN["CSV Generators & Utilities"]:::core
    end

    %% Storage
    CACHESTORE["Local Disk Cache"]:::storage

    %% Extraction & Validation Pipeline
    subgraph "Extraction & Validation Pipeline"
        EXT1["SycamoreExtract.py"]:::extract
        VAL["ValidateRegistrationData.py"]:::extract
        FLT["FilterAndProjectCSVFiles.py"]:::extract
        AATG["AatgConvert.py"]:::extract
        EFR["ExtractForRegistration.py"]:::extract
        MERGE["MergeCSVFiles.py"]:::extract
    end

    %% Format Exporters
    subgraph "Format Exporters"
        SDS["SDS21 Exporter"]:::exporter
        CVR["Clever Exporter"]:::exporter
    end

    %% Output Sinks
    outv21["outv21 (SDS CSVs)"]:::output
    outClever["outClever (Clever CSVs)"]:::output

    %% PowerShell Automation
    subgraph "PowerShell Automation"
        PS1["create-students.ps1"]:::ps
        PS2["delete-groups.ps1"]:::ps
        PS3["set-students-accountenabled.ps1"]:::ps
        PS4["set-students-password.ps1"]:::ps
    end

    %% Data Flows
    SDS1 --> REST
    SDS2 --> REST
    SDS3 --> REST
    CLEVER --> REST
    REST -->|"API requests"| SYCAPI
    SYCAPI -->|"JSON data"| REST
    REST -->|"cache writes"| CACHEM
    CACHEM -->|"files"| CACHESTORE

    SDS1 --> EXT1
    SDS2 --> EXT1
    SDS3 --> EXT1
    CLEVER --> EXT1

    EXT1 --> VAL --> FLT --> AATG --> EFR --> MERGE
    MERGE --> SDS
    MERGE --> CVR

    SDS --> outv21
    CVR --> outClever

    PS1 -->|"reads CSV"| outv21
    PS2 -->|"reads CSV"| outv21
    PS3 -->|"reads CSV"| outv21
    PS4 -->|"reads CSV"| outv21
    PS1 --> AD
    PS2 --> AD
    PS3 --> AD
    PS4 --> AD

    %% Click Events
    click SDS1 "https://github.com/it-gssb/sycamore/blob/master/src/sds/ExtractFromSycamoreToSDS21.py"
    click SDS2 "https://github.com/it-gssb/sycamore/blob/master/src/sds/ExtractFromSycamoreToSDSClever.py"
    click SDS3 "https://github.com/it-gssb/sycamore/blob/master/src/sds/ExtractStudents.py"
    click CLEVER "https://github.com/it-gssb/sycamore/blob/master/src/clever/CleverTransformer.py"
    click REST "https://github.com/it-gssb/sycamore/blob/master/src/lib/SycamoreRest.py"
    click CACHEM "https://github.com/it-gssb/sycamore/blob/master/src/lib/SycamoreCache.py"
    click ENT "https://github.com/it-gssb/sycamore/blob/master/src/lib/SycamoreEntity.py"
    click GEN "https://github.com/it-gssb/sycamore/blob/master/src/lib/Generators.py"
    click EXT1 "https://github.com/it-gssb/sycamore/blob/master/src/extract/SycamoreExtract.py"
    click VAL "https://github.com/it-gssb/sycamore/blob/master/src/extract/ValidateRegistrationData.py"
    click FLT "https://github.com/it-gssb/sycamore/blob/master/src/extract/FilterAndProjectCSVFiles.py"
    click AATG "https://github.com/it-gssb/sycamore/blob/master/src/extract/AatgConvert.py"
    click EFR "https://github.com/it-gssb/sycamore/blob/master/src/extract/ExtractForRegistration.py"
    click MERGE "https://github.com/it-gssb/sycamore/blob/master/src/extract/MergeCSVFiles.py"
    click PS1 "https://github.com/it-gssb/sycamore/blob/master/src/powershell/create-students.ps1"
    click PS2 "https://github.com/it-gssb/sycamore/blob/master/src/powershell/delete-groups.ps1"
    click PS3 "https://github.com/it-gssb/sycamore/blob/master/src/powershell/set-students-accountenabled.ps1"
    click PS4 "https://github.com/it-gssb/sycamore/blob/master/src/powershell/set-students-password.ps1"

    %% Styles
    classDef external fill:#D8BFD8,stroke:#9370DB,stroke-width:2px
    classDef cli fill:#E0FFFF,stroke:#5F9EA0,stroke-width:2px
    classDef core fill:#ADD8E6,stroke:#1E90FF,stroke-width:2px
    classDef storage fill:#F0F8FF,stroke:#4682B4,stroke-width:2px
    classDef extract fill:#98FB98,stroke:#2E8B57,stroke-width:2px
    classDef exporter fill:#FFDAB9,stroke:#FF8C00,stroke-width:2px
    classDef output fill:#FFFACD,stroke:#DAA520,stroke-width:2px
    classDef ps fill:#E6E6FA,stroke:#9370DB,stroke-width:2px
```
