How iscc/iscc-sct works
Project Type and Purpose
• This is a Python library and CLI tool (iscc-sct) that implements the ISCC Semantic Text-Code algorithm.
• It also includes a Gradio-based demo UI and test suite.
Top-Level Components
• CLI Entrypoint (sct command)
• Demo GUI Entrypoint (app.py / Gradio)
• Core Library (iscc_sct/)
• text chunking & embedding
• vector aggregation & binarization
• ISCC encoding
• configuration & utilities
• models & tokenizer assets
• External Dependencies & Services
• Sentence-Transformers model downloads (HuggingFace)
• Tests & Documentation
Architectural Pattern
• Modular pipeline:
Text Input → Chunking → Embedding → Aggregation → Binarization → Encoding → Output
• Separation of concerns:
• Entrypoints (CLI, GUI)
• Core processing library
• Config / options layer
• Persistence of model assets (tokenizer.json)
Relationships & Data Flow
a. CLI / GUI modules invoke the public API (iscc_sct.create())
↳ Pass text and options (bits, granular, debug)
b. Options loader (options.py) reads environment or flags
c. Chunker (utils.py, maybe code_semantic_text.py) splits text → token batches
d. Embedding generator (models.py) loads pre-trained model from HuggingFace
e. Aggregator & Binarizer (in code_semantic_text.py) converts embeddings to binary feature vector
f. ISCC Encoder (in models.py or main.py) prefixes & base32-encodes the binary vector
g. Output printed/formatted by CLI or returned to caller
Key Technologies
• Python 3.10+ packaging (pyproject.toml, Poetry)
• argparse / click style CLI in cli.py
• Gradio for web demo (demo.py, app.py, space.yml)
• SentenceTransformers (HuggingFace) for embeddings
• Base32 encoding & binary operations for ISCC