How brianmg/voynich-nlp-analysis works
Project type and purpose
• This is an open-source computational linguistics/data analysis pipeline built in Python.
• Its goal is to apply modern NLP techniques (SBERT clustering, POS inference, Markov transitions, lexicon hypothesis) to the Voynich Manuscript data.
• It reads raw transcriptions, preprocesses them, clusters root forms, infers grammatical roles, builds transition models, and exports a candidate lexicon.
File structure
• Top-level directories:
• data/ – raw inputs (DOCX, per-folio text files, CSV/JSON lookup tables)
• scripts/ – modular Python scripts for each pipeline stage
• results/ – generated charts, CSV summaries, heatmaps
• requirements.txt – Python dependency manifest
• Each script corresponds to one logical transformation step:
• cluster_roots.py → SBERT embedding + suffix stripping + clustering
• map_lines_to_clusters.py → maps each line of text to a cluster sequence
• pos_model.py → statistical inference of grammatical roles
• transition_matrix.py → builds and visualizes Markov transition matrix
• lexicon_builder.py → section-aware lexicon hypothesis
• cluster_language_similarity.py → optional cross-language comparison
• Data artifacts (CSV, JSON) flow from data/ → scripts/ → results/.
Main components and relationships
• Data Layer
• Raw transcript files (data/voynichese/*.txt, .docx)
• Precomputed lookups (stripped_cluster_lookup.json, unique_stripped_words.json)
• Preprocessing & Feature Extraction
• Suffix stripper
• SBERT embedding engine (external model)
• Clustering Engine
• scikit-learn or UMAP/PCA for dimension reduction
• Mapping Module
• Associates raw lines to cluster IDs
• POS Inference Module
• Statistical modeling of cluster usage patterns
• Transition Matrix Module
• Builds Markov transition matrix, generates heatmap
• Lexicon Builder Module
• Aggregates cluster-word candidates by section and role
• Visualization & Export
• Generates figures (Figure_1.png, heatmap), summary CSVs
• Pipeline Orchestrator (implicit)
• A user may call scripts in series (README outlines “run each stage”)
Important architectural patterns and design principles
• ETL-style, linear pipeline: each script reads standardized inputs and writes standardized outputs.
• Modularity: each analysis step is decoupled in its own script.
• Data-driven: nearly all logic parameterized by input files in data/.
• Extensibility points:
• Swap or retrain SBERT model
• Replace suffix-stripping heuristic
• Or reorder pipeline stages
• Reproducibility: environment pinned in requirements.txt, clear run order in README.
Relevant technologies and libraries
• Python 3.x, pip
• numpy, pandas for data handling
• sentence-transformers (SBERT) for embeddings
• scikit-learn for clustering, PCA
• seaborn/matplotlib for visualizations
• JSON/CSV for intermediate artifacts
Detail
• Script as its own component.
• Intermediate files produced/consumed between stages.