# maxtheyeeter/caesar-stylometry architecture

> Architecture diagram of the GitHub repository maxtheyeeter/caesar-stylometry, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/maxtheyeeter/caesar-stylometry
- Repository: https://github.com/maxtheyeeter/caesar-stylometry

## Overview

1. This repository is a reproducible digital-humanities stylometry pipeline for testing whether Caesar composed *De Bello Gallico* Books I–VII sequentially over time or in one late bulk composition. The architecture is centered on a staged text-processing workflow that turns Perseus XML into multiple parallel corpus representations, then runs several independent style analyses.

2. The primary inputs are two TEI/XML source texts in `data/raw/perseus/`: `caes_bg_lat.xml` (*De Bello Gallico*) and `caes.bc_lat.xml` (*De Bello Civili*). DBG is the target corpus; DBC is the later Caesar anchor used as a dated stylistic reference; Book VIII of DBG (Hirtius) is a built-in negative control.

3. The preprocessing subsystem is the first major boundary. Scripts `01_parse_perseus_xml.py` and `02_parse_tei.py` extract and clean the Perseus markup, `03_build_corpus.py` assembles book- and chapter-level units, and `04_normalize.py` / `05_lemmatize.py` create alternate lexical views. The output of this stage lives under `data/corpus/`, `data/processed/`, and the `data/stylo_corpus_*` folders.

4. The corpus is deliberately split into two granularity layers: books and chapters. That gives two analysis perspectives: coarse book-level trends for chronology and finer chapter-level diagnostics for within-book variation and robustness. The chapter-level exports are especially important for Book VIII separation and outlier-rate testing.

5. There are two lexical representations throughout the pipeline: normalized tokens and lemmas. Normalized tokens are lowercased and orthographically regularized; lemmas are produced with CLTK/Stanza-based lemmatization. This dual representation is an explicit architectural boundary so that the same analyses can be re-run against different linguistic abstractions.

6. Feature generation is the next subsystem. `06_diagnostics.py` validates corpus quality, `07_features_words.py` builds function-word and most-frequent-word matrices, and `08_features_ngrams.py` builds character n-gram matrices. These features are exported per representation and granularity, creating the `outputs/features_*_books.csv` and `outputs/features_*_chapters.csv` artifacts.

7. A separate export path feeds R-based stylometry. `09_export_for_stylo.py` prepares plain-text corpora for `scripts/10_stylo_delta.R`, which is the main boundary between Python preprocessing and R classical stylometry. This R step produces Delta-based distance and clustering outputs that complement the Python analyses and act as a methodological cross-check.

8. The analysis layer is split into several focused scripts rather than one monolith. `11_delta_python.py` computes Python-side distance and classification reports, `12_pca_umap.py` performs unsupervised ordination, `13_dbc_anchor.py` measures distance trends relative to DBC, and `14_drift_tests.py` tests chronological drift patterns. These scripts consume the same feature matrices but answer different statistical questions.

9. Robustness and validity are isolated into their own stage. `15_robustness.py` reruns the key analyses across many condition combinations: lexical level, feature family, MFW cutoff, distance metric, and excursus removal. `16_latinbert.py` adds a semantic embedding cross-check using a Latin BERT model; it is explicitly treated as a comparison point rather than the main evidence line.

10. The final reporting subsystem is `17_report.py`, which aggregates analysis tables, figures, and narrative conclusions into `outputs/REPORT.md`. Supporting summary artifacts such as `outputs/RESULTS_MANIFEST.md`, `outputs/robustness_summary.csv`, and `outputs/book8_separation_report.csv` make the results auditable without rerunning the full pipeline.

11. Pipeline orchestration is handled by `run_all.sh` and `scripts/run_analysis_pipeline.py`. The overall execution order is linear, but the architecture is stage-based: corpus construction, feature extraction, validation, analysis, robustness, cross-check, reporting. This keeps the data flow clear and allows reruns from intermediate checkpoints.

12. The repository is also organized as a data product. `data/raw/` is source-controlled only as a location placeholder, `data/corpus/` and `data/processed/` hold generated intermediate corpora, `figures/` stores the derived visualizations, and `logs/` records per-stage runtime output and diagnostics. The tracked CSVs and PDFs in `outputs/` and `figures/` represent the final analytical surface of the project.

13. The key external dependencies materially shaping the architecture are Python for preprocessing/analysis, R with `stylo` for Delta stylometry, and CLTK for Latin linguistic processing. The Latin BERT model is the only substantive ML component, but it is compartmentalized so it cannot influence the classical stylometry pipeline.

14. In diagram terms, the important data flows are: Perseus XML in → cleaned corpus units → normalized/lemmatized corpora → feature matrices and stylo exports → parallel analysis branches → consolidated results/report. The most important boundaries are raw vs generated data, Python vs R analysis, and stylistic-feature methods vs semantic cross-checks.

Main entry points: [`scripts/17_report.py`](https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/17_report.py) (Report), [`scripts/07_features_words.py`](https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/07_features_words.py) (Word features), [`data/corpus`](https://github.com/maxtheyeeter/caesar-stylometry/tree/main/data/corpus) (Corpus views), [`scripts/run_analysis_pipeline.py`](https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/run_analysis_pipeline.py) (Pipeline runner)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g_inputs["Raw sources"]
  node_raw_perseus[("Perseus XML<br/>tei/xml sources")]
end

subgraph group_g_prep["Preprocessing"]
  node_parse_xml["Parse XML<br/>parser"]
  node_parse_tei["Parse TEI<br/>tei cleaner<br/>[02_parse_tei.py]"]
  node_build_corpus["Build corpus<br/>segmenter<br/>[03_build_corpus.py]"]
  node_normalize["Normalize<br/>orthographic normalization<br/>[04_normalize.py]"]
  node_lemmatize["Lemmatize<br/>[05_lemmatize.py]"]
  node_corpus_views[("Corpus views<br/>generated corpora")]
end

subgraph group_g_features["Feature building"]
  node_diagnostics["Diagnostics<br/>validation<br/>[06_diagnostics.py]"]
  node_word_features["Word features<br/>function-word / mfw matrices"]
  node_ngram_features["N-grams"]
  node_stylo_export["Stylo export<br/>r corpus export"]
end

subgraph group_g_analysis["Analysis"]
  node_r_stylo["R Delta<br/>R/stylo analysis<br/>[10_stylo_delta.R]"]
  node_python_delta["Python Delta<br/>distance / classification<br/>[11_delta_python.py]"]
  node_ordination["Ordination<br/>PCA / UMAP<br/>[12_pca_umap.py]"]
  node_dbc_anchor["DBC anchor<br/>trend analysis<br/>[13_dbc_anchor.py]"]
  node_drift_tests["Drift tests<br/>chronology tests<br/>[14_drift_tests.py]"]
  node_robustness["Robustness<br/>sensitivity analysis<br/>[15_robustness.py]"]
  node_latinbert["Latin BERT<br/>semantic cross-check<br/>[16_latinbert.py]"]
end

subgraph group_g_reporting["Reporting"]
  node_report["Report<br/>result synthesis<br/>[17_report.py]"]
  node_outputs[("Outputs<br/>published artifacts")]
end

subgraph group_g_orchestration["Orchestration"]
  node_pipeline["Pipeline runner<br/>orchestrator"]
  node_run_all["Shell runner<br/>entrypoint<br/>[run_all.sh]"]
end

node_raw_perseus -->|"source XML"| node_parse_xml
node_parse_xml -->|"clean markup"| node_parse_tei
node_parse_tei -->|"text units"| node_build_corpus
node_build_corpus -->|"book/chapter CSVs"| node_normalize
node_build_corpus -->|"book/chapter CSVs"| node_lemmatize
node_normalize -->|"normalized corpora"| node_corpus_views
node_lemmatize -->|"lemma corpora"| node_corpus_views
node_corpus_views -->|"quality check"| node_diagnostics
node_corpus_views -->|"token/lemma views"| node_word_features
node_corpus_views -->|"token/lemma views"| node_ngram_features
node_word_features -->|"feature selection"| node_stylo_export
node_stylo_export -->|"plain text corpora"| node_r_stylo
node_word_features -->|"feature matrices"| node_python_delta
node_ngram_features -->|"feature matrices"| node_ordination
node_word_features -->|"feature matrices"| node_dbc_anchor
node_word_features -->|"feature matrices"| node_drift_tests
node_word_features -->|"analysis inputs"| node_robustness
node_ngram_features -->|"analysis inputs"| node_robustness
node_corpus_views -->|"corpus text"| node_latinbert
node_python_delta -->|"results"| node_report
node_ordination -->|"figures"| node_report
node_dbc_anchor -->|"trend tables"| node_report
node_drift_tests -->|"drift evidence"| node_report
node_robustness -->|"sensitivity tables"| node_report
node_latinbert -->|"cross-check"| node_report
node_r_stylo -->|"delta cross-check"| node_report
node_report -->|"published artifacts"| node_outputs
node_pipeline -->|"stage 1"| node_parse_xml
node_pipeline -->|"stage 2"| node_diagnostics
node_pipeline -->|"stage 3"| node_python_delta
node_pipeline -->|"stage 4"| node_report
node_run_all -->|"entrypoint"| node_pipeline

click node_raw_perseus "https://github.com/maxtheyeeter/caesar-stylometry/tree/main/data/raw/perseus"
click node_parse_xml "https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/01_parse_perseus_xml.py"
click node_parse_tei "https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/02_parse_tei.py"
click node_build_corpus "https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/03_build_corpus.py"
click node_normalize "https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/04_normalize.py"
click node_lemmatize "https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/05_lemmatize.py"
click node_corpus_views "https://github.com/maxtheyeeter/caesar-stylometry/tree/main/data/corpus"
click node_diagnostics "https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/06_diagnostics.py"
click node_word_features "https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/07_features_words.py"
click node_ngram_features "https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/08_features_ngrams.py"
click node_stylo_export "https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/09_export_for_stylo.py"
click node_r_stylo "https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/10_stylo_delta.R"
click node_python_delta "https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/11_delta_python.py"
click node_ordination "https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/12_pca_umap.py"
click node_dbc_anchor "https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/13_dbc_anchor.py"
click node_drift_tests "https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/14_drift_tests.py"
click node_robustness "https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/15_robustness.py"
click node_latinbert "https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/16_latinbert.py"
click node_report "https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/17_report.py"
click node_outputs "https://github.com/maxtheyeeter/caesar-stylometry/tree/main/outputs"
click node_pipeline "https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/run_analysis_pipeline.py"
click node_run_all "https://github.com/maxtheyeeter/caesar-stylometry/blob/main/run_all.sh"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_raw_perseus toneBlue
class node_parse_xml,node_parse_tei,node_build_corpus,node_normalize,node_lemmatize,node_corpus_views toneAmber
class node_diagnostics,node_word_features,node_ngram_features,node_stylo_export toneMint
class node_r_stylo,node_python_delta,node_ordination,node_dbc_anchor,node_drift_tests,node_robustness,node_latinbert toneRose
class node_report,node_outputs toneIndigo
class node_pipeline,node_run_all toneTeal
```

## Components

### Raw sources

Perseus XML inputs and source placeholders.

- [Perseus XML](https://github.com/maxtheyeeter/caesar-stylometry/tree/main/data/raw/perseus): tei/xml sources · Primary Caesar and Cicero raw texts in Perseus/TEI form. · `data/raw/perseus`

### Preprocessing

Parse, clean, segment, normalize, and lemmatize corpora.

- [Parse XML](https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/01_parse_perseus_xml.py): parser · Extracts and cleans Perseus markup. · `scripts/01_parse_perseus_xml.py`
- [Parse TEI](https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/02_parse_tei.py): tei cleaner · Secondary TEI normalization and text cleaning. · `scripts/02_parse_tei.py`
- [Build corpus](https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/03_build_corpus.py): segmenter · Assembles book- and chapter-level corpus units. · `scripts/03_build_corpus.py`
- [Normalize](https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/04_normalize.py): orthographic normalization · Creates lowercased, regularized token views. · `scripts/04_normalize.py`
- [Lemmatize](https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/05_lemmatize.py): latin lemmatizer · Builds lemma corpora using CLTK/Stanza. · `scripts/05_lemmatize.py`
- [Corpus views](https://github.com/maxtheyeeter/caesar-stylometry/tree/main/data/corpus): generated corpora · Book and chapter CSV corpora in token, normalized, and lemmatized forms. · `data/corpus`

### Feature building

Build matrices and exports used by stylometry analyses.

- [Diagnostics](https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/06_diagnostics.py): validation · Checks corpus quality and lemma coverage. · `scripts/06_diagnostics.py`
- [Word features](https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/07_features_words.py): function-word / mfw matrices · Builds function-word and most-frequent-word feature tables. · `scripts/07_features_words.py`
- [N-grams](https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/08_features_ngrams.py): character n-grams · Builds character n-gram feature matrices. · `scripts/08_features_ngrams.py`
- [Stylo export](https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/09_export_for_stylo.py): r corpus export · Prepares plain-text corpora for classical R stylometry. · `scripts/09_export_for_stylo.py`

### Analysis

Parallel statistical and comparative analysis branches.

- [R Delta](https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/10_stylo_delta.R): R/stylo analysis · Classical Delta distance and clustering in R. · `scripts/10_stylo_delta.R`
- [Python Delta](https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/11_delta_python.py): distance / classification · Python-side distances and predictive reports. · `scripts/11_delta_python.py`
- [Ordination](https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/12_pca_umap.py): PCA / UMAP · Unsupervised projection of stylistic space. · `scripts/12_pca_umap.py`
- [DBC anchor](https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/13_dbc_anchor.py): trend analysis · Measures Caesar drift relative to De Bello Civili. · `scripts/13_dbc_anchor.py`
- [Drift tests](https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/14_drift_tests.py): chronology tests · Tests sequential drift across DBG books and chapters. · `scripts/14_drift_tests.py`
- [Robustness](https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/15_robustness.py): sensitivity analysis · Reruns analyses across lexical, feature, metric, and exclusion settings. · `scripts/15_robustness.py`
- [Latin BERT](https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/16_latinbert.py): semantic cross-check · Embedding-based comparison treated as a secondary evidence line. · `scripts/16_latinbert.py`

### Reporting

Consolidated outputs, figures, and narrative results.

- [Report](https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/17_report.py): result synthesis · Aggregates figures, tables, and conclusions into the final report. · `scripts/17_report.py`
- [Outputs](https://github.com/maxtheyeeter/caesar-stylometry/tree/main/outputs): published artifacts · Final CSVs, PDFs, and manifest/report artifacts. · `outputs`

### Orchestration

Top-level runners and pipeline coordination.

- [Pipeline runner](https://github.com/maxtheyeeter/caesar-stylometry/blob/main/scripts/run_analysis_pipeline.py): orchestrator · Coordinates the staged execution order. · `scripts/run_analysis_pipeline.py`
- [Shell runner](https://github.com/maxtheyeeter/caesar-stylometry/blob/main/run_all.sh): entrypoint · Top-level shell entrypoint for the full workflow. · `run_all.sh`

## Connections

- Perseus XML → Parse XML: source XML · Perseus TEI feeds the parser.
- Parse XML → Parse TEI: clean markup · Parsed XML is cleaned and standardized.
- Parse TEI → Build corpus: text units · Cleaned text is segmented into books and chapters.
- Build corpus → Normalize: book/chapter CSVs · Corpus units are normalized into alternate token views.
- Build corpus → Lemmatize: book/chapter CSVs · Corpus units are lemmatized for parallel analysis.
- Normalize → Corpus views: normalized corpora · Normalized CSV corpus views are stored.
- Lemmatize → Corpus views: lemma corpora · Lemmatized CSV corpus views are stored.
- Corpus views → Diagnostics: quality check · Generated corpora are validated before feature extraction.
- Corpus views → Word features: token/lemma views · Corpus views feed word-based features.
- Corpus views → N-grams: token/lemma views · Corpus views feed character n-gram features.
- Word features → Stylo export: feature selection · Word-based corpora are also exported for R stylometry.
- Stylo export → R Delta: plain text corpora · Exports become input to stylo Delta analysis.
- Word features → Python Delta: feature matrices · Feature tables drive Python distance and classification.
- N-grams → Ordination: feature matrices · N-gram matrices drive ordination.
- Word features → DBC anchor: feature matrices · Word features are compared against the DBC anchor.
- Word features → Drift tests: feature matrices · Word features are tested for chronological drift.
- Word features → Robustness: analysis inputs · Core word features are reused under varied settings.
- N-grams → Robustness: analysis inputs · N-gram features are included in sensitivity checks.
- Corpus views → Latin BERT: corpus text · Text corpora support the semantic comparison branch.
- Python Delta → Report: results · Python metrics contribute to the final report.
- Ordination → Report: figures · Ordination outputs inform the report narrative.
- DBC anchor → Report: trend tables · DBC-relative trends are summarized in reporting.
- Drift tests → Report: drift evidence · Chronology tests are summarized in reporting.
- Robustness → Report: sensitivity tables · Robustness sweeps feed the final summary.
- Latin BERT → Report: cross-check · Semantic embeddings are reported as a comparison line.
- R Delta → Report: delta cross-check · R Delta outputs are folded into the report.
- Report → Outputs: published artifacts · The final report and supporting artifacts are written to outputs.
- Pipeline runner → Parse XML: stage 1 · Orchestrates preprocessing start.
- Pipeline runner → Diagnostics: stage 2 · Orchestrates validation and feature generation.
- Pipeline runner → Python Delta: stage 3 · Orchestrates main analysis branches.
- Pipeline runner → Report: stage 4 · Orchestrates consolidation and reporting.
- Shell runner → Pipeline runner: entrypoint · Shell script launches the pipeline runner.