# usnistgov/sctk architecture

> Architecture diagram of the GitHub repository usnistgov/sctk, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/usnistgov/sctk
- Repository: https://github.com/usnistgov/sctk

## Overview

##### Project type and purpose
- SCTK is a monolithic, makefile-driven toolkit of CLI programs for scoring automatic speech recognition output (tools include sclite, asclite, sc_stats, rover, plus many Perl/shell utilities).
- It’s not a web or full-stack app: it’s a collection of executable components, libraries, scripts, documentation, and test suites.

##### File structure
- Top-level directories:
– src/ : source code subdirectories, one per tool or module (sclite, asclite, sc_stats, rover, plus dozens of filter/convert scripts).
– doc/ : documentation and manuals.
– test_suite & src/*/test : test data and self‐tests.
– Makefiles and Dockerfile at root.
- Each src/<tool> folder typically contains:
– C or C++ core modules (.c/.cpp + headers)
– Perl or shell wrapper scripts (.pl, .sh)
– makefile.in or config scripts for building that component.

##### README for additional insights
- Build process: `make config` (invokes src/sclite/config.sh), `make all` (compiles every tool), `make check` (runs tests), and `make install`.
- External dependencies (optional): GNU diffutils, CMU‐Cambridge SLM toolkit for language models, SGML parsers (OpenSP).
- Runtime flows: user invokes a CLI tool (e.g., sclite), passing reference and hypothesis files; sc_stats consumes sclite’s SGML output for statistical tests; other scripts convert and filter formats.
- Dockerfile wraps the entire build+runtime environment for cross‐platform usage.

##### Main components and their relationships
a. Build System
– config.sh & root/Makefile → generates per-component Makefiles → compiles C/C++, installs binaries under bin/
b. CLI Tools Suite
– sclite (C core + optional diff / SLM)
– asclite (C++ core library + wrapper scripts)
– sc_stats (Perl script)
– rover (Perl script)
– dozens of converter/filter scripts (ctmValidator, rttmValidator, align2html, etc.)
c. Core Libraries & Engines
– C/C++ modules under src/sclite and src/asclite/core
– Shared utility code (Levenshtein, graph algorithms, input parsers)
d. Scripts & Wrappers
– Perl and shell scripts in each src/*
– Orchestrate parsing of CTM, STM, RTTM, generation of SGML reports
e. Test Harness
– src/*/test and test_suite/ directories contain unit, regression, and example data tests
– “make check” drives them
f. External Services & Dependencies
– GNU diff (optional alignment method)
– CMU SLM for language‐model‐based word weighting
– SGML parser (nsgmls/OpenSP) for utf_filt

##### Architectural patterns and design principles
- Modular monorepo: each tool is self-contained under src/<tool>.
- Separation of concerns:
– Build/config layer vs. core engine layer vs. wrapper/orchestration layer vs. test layer
- CLI façade over lower-level libraries (C/C++ or Perl modules).
- Data-pipeline style at runtime: input formats → tool → output formats → next tool.

- Components by layers or domains:
– BUILD LAYER (Makefiles, config.sh, Dockerfile)
– SOURCE LAYER (src folders by tool)
– RUNTIME LAYER (executables under bin/; scripts)
– TEST & DOC LAYER (doc/, test_suite/)
– EXTERNAL DEPENDENCIES (diffutils, SLM, OpenSP)
– Blue rectangles for build components
– Green ovals for executables/scripts
– Yellow rectangles for core libraries

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Monorepo Root
    subgraph "Monorepo Root"
        Makefile["Makefile"]:::build
        Dockerfile["Dockerfile"]:::build
    end

    click Makefile "https://github.com/usnistgov/sctk/tree/master/./makefile"
    click Dockerfile "https://github.com/usnistgov/sctk/tree/master/./Dockerfile"

    %% Build Layer
    subgraph "BUILD LAYER"
        ConfigSH["config.sh"]:::build
        RootMake["Makefile → per-component Makefiles"]:::build
    end

    click ConfigSH "https://github.com/usnistgov/sctk/blob/master/./src/sclite/config.sh"

    Makefile -->|"invoke"| ConfigSH
    ConfigSH -->|"generate"| RootMake
    RootMake -->|"compile"| CompileTargets["Compilation Targets"]:::build
    CompileTargets -->|"output bins"| BinDir["bin/ Executables"]:::runtime

    %% SOURCE LAYER
    subgraph "SOURCE LAYER"
        subgraph "sclite"
            ScliteSrc["src/sclite/"]:::source
            ScliteMF["makefile.in"]:::build
            ScliteLib["levenshtein.c, graph.c, sgml.c,..."]:::library
            ScStatsC["sc_stats.c"]:::source
            RoverC["rover.c"]:::source
        end

        subgraph "asclite"
            AscliteSrc["src/asclite/core/"]:::source
            AscliteMF["makefile"]:::build
            AscliteConfig["config.sh"]:::build
        end

        subgraph "Scripts & Converters"
            CTMVal["ctmValidator.pl"]:::runtime
            RTTMVal["rttmValidator.pl"]:::runtime
            STMVal["stmValidator-main.pl"]:::runtime
            UTFFilt["utf_filt.pl"]:::runtime
            AlignHTML["align2html-main.pl"]:::runtime
        end
    end

    click ScliteSrc "https://github.com/usnistgov/sctk/tree/master/./src/sclite/"
    click ScliteMF "https://github.com/usnistgov/sctk/blob/master/./src/sclite/makefile.in"
    click ScStatsC "https://github.com/usnistgov/sctk/blob/master/./src/sclite/sc_stats.c"
    click RoverC "https://github.com/usnistgov/sctk/blob/master/./src/sclite/rover.c"
    click AscliteSrc "https://github.com/usnistgov/sctk/tree/master/./src/asclite/core/"
    click AscliteMF "https://github.com/usnistgov/sctk/tree/master/./src/asclite/makefile"
    click AscliteConfig "https://github.com/usnistgov/sctk/blob/master/./src/asclite/core/config.sh"
    click CTMVal "https://github.com/usnistgov/sctk/blob/master/./src/ctmValidator/ctmValidator.pl"
    click RTTMVal "https://github.com/usnistgov/sctk/blob/master/./src/rttmValidator/rttmValidator.pl"
    click STMVal "https://github.com/usnistgov/sctk/blob/master/./src/stmValidator/stmValidator-main.pl"
    click UTFFilt "https://github.com/usnistgov/sctk/blob/master/./src/utf_filt/utf_filt.pl"
    click AlignHTML "https://github.com/usnistgov/sctk/blob/master/./src/align2html/align2html-main.pl"

    %% Build connections
    RootMake -->|build src| ScliteMF
    RootMake -->|build src| AscliteMF

    ScliteMF -->|compile core| ScliteSrc
    AscliteMF -->|compile core| AscliteSrc

    %% RUNTIME LAYER
    subgraph "RUNTIME LAYER"
        User["User CLI"]:::actor
        ScliteExe["sclite (bin/)"]:::runtime
        AscliteExe["asclite (bin/)"]:::runtime
        ScStatsExe["sc_stats (bin/)"]:::runtime
        RoverExe["rover (bin/)"]:::runtime
    end

    User -->|"CTM/STM in"| ScliteExe
    ScliteExe -->|"SGML out"| ScStatsExe
    ScStatsExe -->|"stats report"| RoverExe

    ScStatsExe -->|"uses"| ScStatsC
    RoverExe -->|"uses"| RoverC
    ScliteExe -->|"links"| ScliteLib
    AscliteExe -->|"links"| AscliteSrc

    %% TEST & DOC LAYER
    subgraph "TEST & DOC LAYER"
        subgraph "Test Suite"
            AscliteTest["src/asclite/test/"]:::test
            ScliteTest["src/sclite/testdata/"]:::test
            GlobalTest["src/test_suite/"]:::test
        end
        subgraph "Documentation"
            DocDir["doc/"]:::doc
        end
    end

    click AscliteTest "https://github.com/usnistgov/sctk/tree/master/./src/asclite/test/"
    click ScliteTest "https://github.com/usnistgov/sctk/tree/master/./src/sclite/testdata/"
    click GlobalTest "https://github.com/usnistgov/sctk/tree/master/./src/test_suite/"
    click DocDir "https://github.com/usnistgov/sctk/tree/master/./doc/"

    %% DOC and TEST connections
    Makefile -->|"make check"| AscliteTest
    Makefile -->|"make check"| ScliteTest
    Makefile -->|"make check"| GlobalTest

    %% EXTERNAL DEPENDENCIES
    subgraph "EXTERNAL DEPENDENCIES"
        Diffutils["GNU diffutils"]:::external
        SLM["CMU SLM toolkit"]:::external
        SGMLParser["SGML parser (OpenSP)"]:::external
    end

    ScliteExe -->|"optional align"| Diffutils
    ScliteExe -->|"LM scoring"| SLM
    UTFFilt -->|"uses DTD"| SGMLParser

    %% Styles
    classDef build fill:#cce5ff,stroke:#3399ff,stroke-width:1px
    classDef source fill:#e6ffe6,stroke:#33cc33,stroke-width:1px
    classDef library fill:#fff5cc,stroke:#e6b800,stroke-width:1px
    classDef runtime fill:#d9d9d9,stroke:#7f7f7f,stroke-width:1px
    classDef actor fill:#f2f2f2,stroke:#bfbfbf,stroke-width:1px,stroke-dasharray: 5 5
    classDef test fill:#f9e6ff,stroke:#d280ff,stroke-width:1px,stroke-dasharray: 2 2
    classDef doc fill:#ffe6e6,stroke:#ff6666,stroke-width:1px,stroke-dasharray: 2 2
    classDef external fill:#eeeeee,stroke:#999999,stroke-width:1px,stroke-dasharray: 4 2
```
