# brianmg/voynich-nlp-analysis architecture

> Architecture diagram of the GitHub repository brianmg/voynich-nlp-analysis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/brianmg/voynich-nlp-analysis
- Repository: https://github.com/brianmg/voynich-nlp-analysis

## Overview

##### Project type and purpose
- This is an open-source computational linguistics/data analysis pipeline built in Python.
- Its goal is to apply modern NLP techniques (SBERT clustering, POS inference, Markov transitions, lexicon hypothesis) to the Voynich Manuscript data.
- It reads raw transcriptions, preprocesses them, clusters root forms, infers grammatical roles, builds transition models, and exports a candidate lexicon.

##### File structure
- Top-level directories:
- data/      – raw inputs (DOCX, per-folio text files, CSV/JSON lookup tables)
- scripts/   – modular Python scripts for each pipeline stage
- results/   – generated charts, CSV summaries, heatmaps
- requirements.txt  – Python dependency manifest
- Each script corresponds to one logical transformation step:
- cluster_roots.py → SBERT embedding + suffix stripping + clustering
- map_lines_to_clusters.py → maps each line of text to a cluster sequence
- pos_model.py → statistical inference of grammatical roles
- transition_matrix.py → builds and visualizes Markov transition matrix
- lexicon_builder.py → section-aware lexicon hypothesis
- cluster_language_similarity.py → optional cross-language comparison
- Data artifacts (CSV, JSON) flow from data/ → scripts/ → results/.

##### Main components and relationships
- Data Layer
- Raw transcript files (data/voynichese/*.txt, .docx)
- Precomputed lookups (stripped_cluster_lookup.json, unique_stripped_words.json)
- Preprocessing & Feature Extraction
- Suffix stripper
- SBERT embedding engine (external model)
- Clustering Engine
- scikit-learn or UMAP/PCA for dimension reduction
- Mapping Module
- Associates raw lines to cluster IDs
- POS Inference Module
- Statistical modeling of cluster usage patterns
- Transition Matrix Module
- Builds Markov transition matrix, generates heatmap
- Lexicon Builder Module
- Aggregates cluster-word candidates by section and role
- Visualization & Export
- Generates figures (Figure_1.png, heatmap), summary CSVs
- Pipeline Orchestrator (implicit)
- A user may call scripts in series (README outlines “run each stage”)

##### Important architectural patterns and design principles
- ETL-style, linear pipeline: each script reads standardized inputs and writes standardized outputs.
- Modularity: each analysis step is decoupled in its own script.
- Data-driven: nearly all logic parameterized by input files in data/.
- Extensibility points:
- Swap or retrain SBERT model
- Replace suffix-stripping heuristic
- Or reorder pipeline stages
- Reproducibility: environment pinned in requirements.txt, clear run order in README.

##### Relevant technologies and libraries
- Python 3.x, pip
- numpy, pandas for data handling
- sentence-transformers (SBERT) for embeddings
- scikit-learn for clustering, PCA
- seaborn/matplotlib for visualizations
- JSON/CSV for intermediate artifacts

##### Detail
- Script as its own component.
- Intermediate files produced/consumed between stages.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    subgraph "Data Layer"
        RawTxt["Raw Transcript Files\n(data/voynitchese/*.txt)"]:::data
        Docx["Document Inputs\n(data/AB.docx)"]:::data
        StrippedLookup["Stripped Cluster Lookup\n(data/stripped_cluster_lookup.json)"]:::data
        UniqueWords["Unique Stripped Words\n(data/unique_stripped_words.json)"]:::data
        ClusterSeq["Line-Cluster Sequences\n(data/voynich_line_clusters.csv)"]:::data
        LexTable1["Lexicon Hypothesis Table\n(data/Lexicon_Hypothesis_Table__Top_50_Words_Section_.csv)"]:::data
        LexTable2["Lexicon Table with Section Exclusivity\n(data/Lexicon_Table_with_Section_Exclusivity.csv)"]:::data
    end

    %% Processing Pipeline
    subgraph "Processing Pipeline"
        ClusterRoots["Suffix Stripper & SBERT Embedder\n(scripts/cluster_roots.py)"]:::processing
        MapLines["Line-to-Cluster Mapper\n(scripts/map_lines_to_clusters.py)"]:::processing
        POSModel["POS Inference Module\n(scripts/pos_model.py)"]:::processing
        TransMatrix["Transition Matrix Builder\n(scripts/transition_matrix.py)"]:::processing
        LexBuilder["Lexicon Hypothesis Builder\n(scripts/lexicon_builder.py)"]:::processing
        ClusterLangSim(["Cross-Language Similarity\n(scripts/cluster_language_similarity.py)"]):::optional
    end

    %% External Services
    SBERT["SBERT Embedding Model"]:::external

    %% Outputs
    subgraph "Results & Outputs"
        ClusterRoleSummary["Cluster-Role Summary\n(results/cluster_role_summary.csv)"]:::data
        TransMatrixCSV["Cluster Transition Matrix\n(results/cluster_transition_matrix.csv)"]:::data
        LexCandidates["Lexicon Candidates\n(results/lexicon_candidates.csv)"]:::data
    end

    %% Orchestration & Reproducibility
    subgraph "Orchestration & Reproducibility"
        README["Pipeline Orchestrator\n(README.md)"]:::processing
        Requirements["Dependencies\n(requirements.txt)"]:::processing
    end

    %% Data Flow
    RawTxt -->|reads| ClusterRoots
    Docx -->|reads| ClusterRoots
    StrippedLookup -->|reads| ClusterRoots
    UniqueWords -->|reads| ClusterRoots
    ClusterRoots -->|writes| ClusterSeq
    ClusterRoots -->|calls| SBERT

    ClusterSeq -->|reads| MapLines
    StrippedLookup -->|reads| MapLines
    MapLines -->|writes| POSModel
    MapLines -->|writes| ClusterSeq
    POSModel -->|writes| ClusterRoleSummary
    POSModel -->|triggers| TransMatrix
    TransMatrix -->|writes| TransMatrixCSV
    POSModel -->|writes| LexBuilder
    LexTable1 -->|reads| LexBuilder
    LexTable2 -->|reads| LexBuilder
    LexBuilder -->|writes| LexCandidates

    ClusterSeq -->|optional input| ClusterLangSim
    StrippedLookup -->|optional input| ClusterLangSim

    %% Orchestration connections
    README -.-> ClusterRoots
    README -.-> MapLines
    README -.-> POSModel
    README -.-> TransMatrix
    README -.-> LexBuilder
    README -.-> ClusterLangSim
    Requirements -.->|pins env| ClusterRoots
    Requirements -.->|pins env| MapLines
    Requirements -.->|pins env| POSModel
    Requirements -.->|pins env| TransMatrix
    Requirements -.->|pins env| LexBuilder
    Requirements -.->|pins env| ClusterLangSim

    %% Click Events
    click RawTxt "https://github.com/brianmg/voynich-nlp-analysis/blob/main/data/voynitchese/*.txt"
    click Docx "https://github.com/brianmg/voynich-nlp-analysis/blob/main/data/AB.docx"
    click StrippedLookup "https://github.com/brianmg/voynich-nlp-analysis/blob/main/data/stripped_cluster_lookup.json"
    click UniqueWords "https://github.com/brianmg/voynich-nlp-analysis/blob/main/data/unique_stripped_words.json"
    click ClusterSeq "https://github.com/brianmg/voynich-nlp-analysis/blob/main/data/voynich_line_clusters.csv"
    click LexTable1 "https://github.com/brianmg/voynich-nlp-analysis/blob/main/data/Lexicon_Hypothesis_Table__Top_50_Words_Section_.csv"
    click LexTable2 "https://github.com/brianmg/voynich-nlp-analysis/blob/main/data/Lexicon_Table_with_Section_Exclusivity.csv"
    click ClusterRoots "https://github.com/brianmg/voynich-nlp-analysis/blob/main/scripts/cluster_roots.py"
    click MapLines "https://github.com/brianmg/voynich-nlp-analysis/blob/main/scripts/map_lines_to_clusters.py"
    click POSModel "https://github.com/brianmg/voynich-nlp-analysis/blob/main/scripts/pos_model.py"
    click TransMatrix "https://github.com/brianmg/voynich-nlp-analysis/blob/main/scripts/transition_matrix.py"
    click LexBuilder "https://github.com/brianmg/voynich-nlp-analysis/blob/main/scripts/lexicon_builder.py"
    click ClusterLangSim "https://github.com/brianmg/voynich-nlp-analysis/blob/main/scripts/cluster_language_similarity.py"
    click README "https://github.com/brianmg/voynich-nlp-analysis/blob/main/README.md"
    click Requirements "https://github.com/brianmg/voynich-nlp-analysis/blob/main/requirements.txt"
    click ClusterRoleSummary "https://github.com/brianmg/voynich-nlp-analysis/blob/main/results/cluster_role_summary.csv"
    click TransMatrixCSV "https://github.com/brianmg/voynich-nlp-analysis/blob/main/results/cluster_transition_matrix.csv"
    click LexCandidates "https://github.com/brianmg/voynich-nlp-analysis/blob/main/results/lexicon_candidates.csv"

    %% Styles
    classDef data fill:#D0E6FF,stroke:#004A7F,stroke-width:1px
    classDef processing fill:#DFFFD0,stroke:#3A7F00,stroke-width:1px
    classDef external fill:#F0D0FF,stroke:#7F004A,stroke-width:1px
    classDef optional fill:#FFD0D0,stroke:#7F0000,stroke-width:1px
```
