How swirepe/ktformatter works
1. Repository purpose
- This is a small command-line data preparation utility, not a service or library.
- Its job is to convert raw CSV-like educational event logs into the sequence format expected by Bayesian Knowledge Tracing and related knowledge-tracing models.
2. Main executable
- The core logic appears to live in a single Python script, KTFormatter.py.
- ktfbridge.bat is a Windows wrapper for launching the formatter from the command line.
3. Primary boundary: raw tabular input vs. model-ready output
- The tool reads a flat file containing student, skill, and response columns.
- It emits one or more transformed output files where records are reorganized for training and evaluation workflows in knowledge tracing.
4. CLI-driven configuration
- The script is controlled entirely through command-line arguments.
- Key required inputs define which columns contain skill IDs, student IDs, and binary responses, plus the input file and output directory.
- This makes the tool generic across datasets rather than tied to a single schema.
5. Input normalization layer
- The formatter supports multiple incoming delimiters: commas, tabs, or spaces.
- It can strip an input header row.
- There is a KDD-specific mode for skill fields containing multiple skill labels separated by ~~, indicating dataset-specific preprocessing is a first-class concern.
6. Transformation layer
- The core transformation is grouping/chopping the raw event stream into knowledge-tracing-friendly sequences.
- It can optionally split the dataset by skill, and more granularly by student.
- It can also partition data into train/test by percentage, which is a separate concern from structural splitting.
7. Identifier canonicalization
- The tool can remap skills and students to integer IDs.
- That suggests a normalization step for downstream algorithms that expect compact categorical identifiers.
- This conversion is optional, so the pipeline can preserve original IDs when desired.
8. Output formatting and serialization
- Output delimiter can be selected independently from input delimiter, again supporting commas, tabs, or spaces.
- Output can be gzip-compressed, which matters for large educational datasets.
- The output directory is the main sink for the generated transformed artifacts.
9. Data flow
- Raw CSV-like file in.
- Parse and optionally remove header.
- Interpret chosen columns for student, skill, and response.
- Optionally normalize IDs and split records by skill/student and train/test.
- Write formatted files to the output directory, optionally compressed.
10. Likely architectural shape
- This repo is essentially a single-purpose batch ETL utility with one processing pipeline and a few format-specific branches.
- There are no visible external services, databases, or web components.
- The main architectural diagram should show a CLI entry point feeding a preprocessing/transformation pipeline that produces multiple output datasets.
11. Domain assumption
- The tool assumes the response column is binary, specifically 0/1, which is important to the data contract.
- That reflects its target domain: educational response logs for Bayesian Knowledge Tracing rather than arbitrary CSV transformation.
12. Practical diagram takeaway
- The important boundaries are: command-line interface, input file parsing, dataset normalization/splitting, and output generation.
- The repository is intentionally minimal, so the architecture is dominated by data flow rather than interacting subsystems.
Main entry points: KTFormatter.py CLI args · ktfbridge.bat Windows wrapper
Components 7
Entry & CLI
Command-line entry and invocation wrapper
- Windows wrapper · Batch file for launching the formatter on Windows
ktfbridge.bat - CLI args · Command-line configuration for input columns, modes, and output settings
KTFormatter.py
Processing Pipeline
Main data preparation flow from raw logs to model-ready sequences
- Normalize data · Handles dataset-specific preprocessing such as KDD multi-skill fields and identifier remapping
KTFormatter.py - Split sequences · Groups event logs into knowledge-tracing sequences and can split by skill, student, or train/test percentage
KTFormatter.py - ID mapping · Optionally remaps student and skill identifiers to compact integer IDs
KTFormatter.py
I/O Formats
Input parsing and output serialization concerns
- Parse input · Reads raw tabular logs with selectable delimiters and optional header stripping
KTFormatter.py - Write output · Emits formatted files to the output directory with selectable delimiters and optional gzip compression
KTFormatter.py
Connections 10
- Windows wrapper → CLI args: launches · Windows wrapper starts the Python formatter
- CLI args → Parse input: configures · CLI arguments determine input file, columns, and parsing mode
- Parse input → Normalize data: feeds · Parsed records enter normalization and dataset-specific cleanup
- Normalize data → ID mapping: optional · Identifiers may be remapped to integer IDs
- Normalize data → Split sequences: routes · Normalized rows are organized into knowledge-tracing sequences
- ID mapping → Split sequences: supports · Canonical IDs can be used before structural splitting
- Split sequences → Write output: produces · Transformed sequences are written as output artifacts
- CLI args → Write output: configures · CLI also controls output delimiter, compression, and destination directory
- CLI args → Normalize data: enables · CLI toggles header handling and KDD-specific preprocessing
- CLI args → Split sequences: enables · CLI toggles split-by-skill, split-by-student, and train/test partitioning