# iscc/iscc-sct architecture

> Architecture diagram of the GitHub repository iscc/iscc-sct, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/iscc/iscc-sct
- Repository: https://github.com/iscc/iscc-sct

## Overview

##### Project Type and Purpose
- This is a Python library and CLI tool (`iscc-sct`) that implements the ISCC Semantic Text-Code algorithm.
- It also includes a Gradio-based demo UI and test suite.

##### Top-Level Components
- CLI Entrypoint (sct command)
- Demo GUI Entrypoint (app.py / Gradio)
- Core Library (`iscc_sct/`)
- text chunking & embedding
- vector aggregation & binarization
- ISCC encoding
- configuration & utilities
- models & tokenizer assets
- External Dependencies & Services
- Sentence-Transformers model downloads (HuggingFace)
- Tests & Documentation

##### Architectural Pattern
- Modular pipeline:
Text Input → Chunking → Embedding → Aggregation → Binarization → Encoding → Output
- Separation of concerns:
- Entrypoints (CLI, GUI)
- Core processing library
- Config / options layer
- Persistence of model assets (tokenizer.json)

##### Relationships & Data Flow
a. CLI / GUI modules invoke the public API (`iscc_sct.create()`)
↳ Pass text and options (bits, granular, debug)
b. Options loader (`options.py`) reads environment or flags
c. Chunker (`utils.py`, maybe `code_semantic_text.py`) splits text → token batches
d. Embedding generator (`models.py`) loads pre-trained model from HuggingFace
e. Aggregator & Binarizer (in `code_semantic_text.py`) converts embeddings to binary feature vector
f. ISCC Encoder (in `models.py` or `main.py`) prefixes & base32-encodes the binary vector
g. Output printed/formatted by CLI or returned to caller

##### Key Technologies
- Python 3.10+ packaging (pyproject.toml, Poetry)
- argparse / click style CLI in `cli.py`
- Gradio for web demo (`demo.py`, `app.py`, `space.yml`)
- SentenceTransformers (HuggingFace) for embeddings
- Base32 encoding & binary operations for ISCC

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interfaces
    subgraph "User Interfaces"
        CLI["CLI Entrypoint (sct command)"]:::frontend
        GUI["Gradio GUI Entrypoint (app.py)"]:::frontend
        DEMOCTRL["Demo GUI Controller (demo.py)"]:::frontend
    end

    %% Configuration Layer
    subgraph "Configuration Layer"
        OPTIONS["Configuration & Options Loader"]:::config
        SAMPLES["Sample Configuration (samples.yml)"]:::config
        SPACECFG["Space Configuration (space.yml)"]:::config
    end

    %% Core Pipeline
    subgraph "Core Pipeline"
        MAIN["Core Public API & Orchestrator (main.py)"]:::core
        CHUNKER["Text Chunking & Utilities (utils.py)"]:::core
        EMBEDDER["Embedding Model Loader & Inference (models.py)"]:::core
        AGGREGATOR["Vector Aggregation"]:::core
        BINARIZER["Vector Binarization"]:::core
        ENCODER["ISCC Encoder"]:::core
        TOKENIZER["Tokenizer Asset (tokenizer.json)"]:::config
    end

    %% External Services
    MODELHUB["HuggingFace Model Hub"]:::external

    %% Output
    OUTPUT["ISCC Code Output"]:::output

    %% Auxiliary
    subgraph "Auxiliary (non-runtime)"
        TESTS["Tests Suite (tests/)"]:::aux
        DOCS["Documentation (docs/)"]:::aux
        DEV["Development Helpers (dev.py)"]:::aux
    end

    %% Data Flow
    CLI -->|"flags & raw text"| OPTIONS
    GUI -->|"flags & raw text"| DEMOCTRL
    DEMOCTRL -->|"passes to options"| OPTIONS
    OPTIONS -->|"configured options"| MAIN

    MAIN -->|"raw text"| CHUNKER
    CHUNKER -->|"token batches"| EMBEDDER
    EMBEDDER -->|"embedding vectors"| AGGREGATOR
    AGGREGATOR -->|"aggregated vector"| BINARIZER
    BINARIZER -->|"binary vector"| ENCODER
    ENCODER -->|"ISCC code"| OUTPUT

    EMBEDDER <-->|"download & cache"| MODELHUB
    TOKENIZER -->|"vocabulary"| EMBEDDER

    %% Click Events
    click CLI "https://github.com/iscc/iscc-sct/blob/main/iscc_sct/cli.py"
    click GUI "https://github.com/iscc/iscc-sct/blob/main/app.py"
    click DEMOCTRL "https://github.com/iscc/iscc-sct/blob/main/iscc_sct/demo.py"
    click MAIN "https://github.com/iscc/iscc-sct/blob/main/iscc_sct/main.py"
    click OPTIONS "https://github.com/iscc/iscc-sct/blob/main/iscc_sct/options.py"
    click CHUNKER "https://github.com/iscc/iscc-sct/blob/main/iscc_sct/utils.py"
    click EMBEDDER "https://github.com/iscc/iscc-sct/blob/main/iscc_sct/models.py"
    click AGGREGATOR "https://github.com/iscc/iscc-sct/blob/main/iscc_sct/code_semantic_text.py"
    click BINARIZER "https://github.com/iscc/iscc-sct/blob/main/iscc_sct/code_semantic_text.py"
    click ENCODER "https://github.com/iscc/iscc-sct/blob/main/iscc_sct/models.py"
    click TOKENIZER "https://github.com/iscc/iscc-sct/blob/main/iscc_sct/tokenizer.json"
    click SAMPLES "https://github.com/iscc/iscc-sct/blob/main/iscc_sct/samples.yml"
    click SPACECFG "https://github.com/iscc/iscc-sct/blob/main/space.yml"
    click TESTS "https://github.com/iscc/iscc-sct/tree/main/tests/"
    click DOCS "https://github.com/iscc/iscc-sct/tree/main/docs/"
    click DEV "https://github.com/iscc/iscc-sct/blob/main/iscc_sct/dev.py"

    %% Styles
    classDef frontend fill:#AED6F1,stroke:#1F618D,color:#1F618D;
    classDef core fill:#ABEBC6,stroke:#229954,color:#145A32;
    classDef config fill:#F9E79F,stroke:#B9770E,color:#7D6608;
    classDef external fill:#F5B7B1,stroke:#C0392B,color:#7B241C;
    classDef output fill:#D2B4DE,stroke:#6C3483,color:#4A235A;
    classDef aux fill:#DDDDDD,stroke:#888888,stroke-dasharray:5 5,color:#555555;
```
