# swirepe/ktformatter architecture

> Architecture diagram of the GitHub repository swirepe/ktformatter, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/swirepe/ktformatter
- Repository: https://github.com/swirepe/ktformatter

## Overview

1. **Repository purpose**
   - This is a small command-line data preparation utility, not a service or library.
   - Its job is to convert raw CSV-like educational event logs into the sequence format expected by Bayesian Knowledge Tracing and related knowledge-tracing models.

2. **Main executable**
   - The core logic appears to live in a single Python script, `KTFormatter.py`.
   - `ktfbridge.bat` is a Windows wrapper for launching the formatter from the command line.

3. **Primary boundary: raw tabular input vs. model-ready output**
   - The tool reads a flat file containing student, skill, and response columns.
   - It emits one or more transformed output files where records are reorganized for training and evaluation workflows in knowledge tracing.

4. **CLI-driven configuration**
   - The script is controlled entirely through command-line arguments.
   - Key required inputs define which columns contain skill IDs, student IDs, and binary responses, plus the input file and output directory.
   - This makes the tool generic across datasets rather than tied to a single schema.

5. **Input normalization layer**
   - The formatter supports multiple incoming delimiters: commas, tabs, or spaces.
   - It can strip an input header row.
   - There is a KDD-specific mode for skill fields containing multiple skill labels separated by `~~`, indicating dataset-specific preprocessing is a first-class concern.

6. **Transformation layer**
   - The core transformation is grouping/chopping the raw event stream into knowledge-tracing-friendly sequences.
   - It can optionally split the dataset by skill, and more granularly by student.
   - It can also partition data into train/test by percentage, which is a separate concern from structural splitting.

7. **Identifier canonicalization**
   - The tool can remap skills and students to integer IDs.
   - That suggests a normalization step for downstream algorithms that expect compact categorical identifiers.
   - This conversion is optional, so the pipeline can preserve original IDs when desired.

8. **Output formatting and serialization**
   - Output delimiter can be selected independently from input delimiter, again supporting commas, tabs, or spaces.
   - Output can be gzip-compressed, which matters for large educational datasets.
   - The output directory is the main sink for the generated transformed artifacts.

9. **Data flow**
   - Raw CSV-like file in.
   - Parse and optionally remove header.
   - Interpret chosen columns for student, skill, and response.
   - Optionally normalize IDs and split records by skill/student and train/test.
   - Write formatted files to the output directory, optionally compressed.

10. **Likely architectural shape**
   - This repo is essentially a single-purpose batch ETL utility with one processing pipeline and a few format-specific branches.
   - There are no visible external services, databases, or web components.
   - The main architectural diagram should show a CLI entry point feeding a preprocessing/transformation pipeline that produces multiple output datasets.

11. **Domain assumption**
   - The tool assumes the response column is binary, specifically 0/1, which is important to the data contract.
   - That reflects its target domain: educational response logs for Bayesian Knowledge Tracing rather than arbitrary CSV transformation.

12. **Practical diagram takeaway**
   - The important boundaries are: command-line interface, input file parsing, dataset normalization/splitting, and output generation.
   - The repository is intentionally minimal, so the architecture is dominated by data flow rather than interacting subsystems.

Main entry points: [`KTFormatter.py`](https://github.com/swirepe/ktformatter/blob/master/KTFormatter.py) (CLI args), [`ktfbridge.bat`](https://github.com/swirepe/ktformatter/blob/master/ktfbridge.bat) (Windows wrapper)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Entry & CLI"]
  node_batch_wrapper["Windows wrapper<br/>launcher<br/>[ktfbridge.bat]"]
  node_cli["CLI args<br/>command-line<br/>[KTFormatter.py]"]
end

subgraph group_pipeline["Processing Pipeline"]
  node_normalize{{"Normalize data<br/>preprocess<br/>[KTFormatter.py]"}}
  node_split_seq{{"Split sequences<br/>transform<br/>[KTFormatter.py]"}}
  node_id_map(("ID mapping<br/>canonicalization<br/>[KTFormatter.py]"))
end

subgraph group_io["I/O Formats"]
  node_input_parse["Parse input<br/>csv-like parser<br/>[KTFormatter.py]"]
  node_output_write["Write output<br/>serializer<br/>[KTFormatter.py]"]
end

node_batch_wrapper -->|"launches"| node_cli
node_cli -->|"configures"| node_input_parse
node_input_parse -->|"feeds"| node_normalize
node_normalize -->|"optional"| node_id_map
node_normalize -->|"routes"| node_split_seq
node_id_map -->|"supports"| node_split_seq
node_split_seq -->|"produces"| node_output_write
node_cli -->|"configures"| node_output_write
node_cli -->|"enables"| node_normalize
node_cli -->|"enables"| node_split_seq

click node_batch_wrapper "https://github.com/swirepe/ktformatter/blob/master/ktfbridge.bat"
click node_cli "https://github.com/swirepe/ktformatter/blob/master/KTFormatter.py"
click node_input_parse "https://github.com/swirepe/ktformatter/blob/master/KTFormatter.py"
click node_normalize "https://github.com/swirepe/ktformatter/blob/master/KTFormatter.py"
click node_split_seq "https://github.com/swirepe/ktformatter/blob/master/KTFormatter.py"
click node_id_map "https://github.com/swirepe/ktformatter/blob/master/KTFormatter.py"
click node_output_write "https://github.com/swirepe/ktformatter/blob/master/KTFormatter.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_batch_wrapper,node_cli toneBlue
class node_normalize,node_split_seq,node_id_map toneAmber
class node_input_parse,node_output_write toneMint
```

## Components

### Entry & CLI

Command-line entry and invocation wrapper

- [Windows wrapper](https://github.com/swirepe/ktformatter/blob/master/ktfbridge.bat): launcher · Batch file for launching the formatter on Windows · `ktfbridge.bat`
- [CLI args](https://github.com/swirepe/ktformatter/blob/master/KTFormatter.py): command-line · Command-line configuration for input columns, modes, and output settings · `KTFormatter.py`

### Processing Pipeline

Main data preparation flow from raw logs to model-ready sequences

- [Normalize data](https://github.com/swirepe/ktformatter/blob/master/KTFormatter.py): preprocess · Handles dataset-specific preprocessing such as KDD multi-skill fields and identifier remapping · `KTFormatter.py`
- [Split sequences](https://github.com/swirepe/ktformatter/blob/master/KTFormatter.py): transform · Groups event logs into knowledge-tracing sequences and can split by skill, student, or train/test percentage · `KTFormatter.py`
- [ID mapping](https://github.com/swirepe/ktformatter/blob/master/KTFormatter.py): canonicalization · Optionally remaps student and skill identifiers to compact integer IDs · `KTFormatter.py`

### I/O Formats

Input parsing and output serialization concerns

- [Parse input](https://github.com/swirepe/ktformatter/blob/master/KTFormatter.py): csv-like parser · Reads raw tabular logs with selectable delimiters and optional header stripping · `KTFormatter.py`
- [Write output](https://github.com/swirepe/ktformatter/blob/master/KTFormatter.py): serializer · Emits formatted files to the output directory with selectable delimiters and optional gzip compression · `KTFormatter.py`

## Connections

- Windows wrapper → CLI args: launches · Windows wrapper starts the Python formatter
- CLI args → Parse input: configures · CLI arguments determine input file, columns, and parsing mode
- Parse input → Normalize data: feeds · Parsed records enter normalization and dataset-specific cleanup
- Normalize data → ID mapping: optional · Identifiers may be remapped to integer IDs
- Normalize data → Split sequences: routes · Normalized rows are organized into knowledge-tracing sequences
- ID mapping → Split sequences: supports · Canonical IDs can be used before structural splitting
- Split sequences → Write output: produces · Transformed sequences are written as output artifacts
- CLI args → Write output: configures · CLI also controls output delimiter, compression, and destination directory
- CLI args → Normalize data: enables · CLI toggles header handling and KDD-specific preprocessing
- CLI args → Split sequences: enables · CLI toggles split-by-skill, split-by-student, and train/test partitioning