maxtheyeeter/caesar-stylometry

1. This repository is a reproducible digital-humanities stylometry pipeline for testing whether Caesar composed *De Bello Gallico* Books I–VII sequentially over time or in one late bulk composition. The architecture is centered on a staged text-processing workflow that turns Perseus XML into multiple parallel corpus representations, then runs several independent style analyses.

2. The primary inputs are two TEI/XML source texts in data/raw/perseus/: caes_bg_lat.xml (*De Bello Gallico*) and caes.bc_lat.xml (*De Bello Civili*). DBG is the target corpus; DBC is the later Caesar anchor used as a dated stylistic reference; Book VIII of DBG (Hirtius) is a built-in negative control.

3. The preprocessing subsystem is the first major boundary. Scripts 01_parse_perseus_xml.py and 02_parse_tei.py extract and clean the Perseus markup, 03_build_corpus.py assembles book- and chapter-level units, and 04_normalize.py / 05_lemmatize.py create alternate lexical views. The output of this stage lives under data/corpus/, data/processed/, and the data/stylo_corpus_* folders.

4. The corpus is deliberately split into two granularity layers: books and chapters. That gives two analysis perspectives: coarse book-level trends for chronology and finer chapter-level diagnostics for within-book variation and robustness. The chapter-level exports are especially important for Book VIII separation and outlier-rate testing.

5. There are two lexical representations throughout the pipeline: normalized tokens and lemmas. Normalized tokens are lowercased and orthographically regularized; lemmas are produced with CLTK/Stanza-based lemmatization. This dual representation is an explicit architectural boundary so that the same analyses can be re-run against different linguistic abstractions.

6. Feature generation is the next subsystem. 06_diagnostics.py validates corpus quality, 07_features_words.py builds function-word and most-frequent-word matrices, and 08_features_ngrams.py builds character n-gram matrices. These features are exported per representation and granularity, creating the outputs/features_*_books.csv and outputs/features_*_chapters.csv artifacts.

7. A separate export path feeds R-based stylometry. 09_export_for_stylo.py prepares plain-text corpora for scripts/10_stylo_delta.R, which is the main boundary between Python preprocessing and R classical stylometry. This R step produces Delta-based distance and clustering outputs that complement the Python analyses and act as a methodological cross-check.

8. The analysis layer is split into several focused scripts rather than one monolith. 11_delta_python.py computes Python-side distance and classification reports, 12_pca_umap.py performs unsupervised ordination, 13_dbc_anchor.py measures distance trends relative to DBC, and 14_drift_tests.py tests chronological drift patterns. These scripts consume the same feature matrices but answer different statistical questions.

9. Robustness and validity are isolated into their own stage. 15_robustness.py reruns the key analyses across many condition combinations: lexical level, feature family, MFW cutoff, distance metric, and excursus removal. 16_latinbert.py adds a semantic embedding cross-check using a Latin BERT model; it is explicitly treated as a comparison point rather than the main evidence line.

10. The final reporting subsystem is 17_report.py, which aggregates analysis tables, figures, and narrative conclusions into outputs/REPORT.md. Supporting summary artifacts such as outputs/RESULTS_MANIFEST.md, outputs/robustness_summary.csv, and outputs/book8_separation_report.csv make the results auditable without rerunning the full pipeline.

11. Pipeline orchestration is handled by run_all.sh and scripts/run_analysis_pipeline.py. The overall execution order is linear, but the architecture is stage-based: corpus construction, feature extraction, validation, analysis, robustness, cross-check, reporting. This keeps the data flow clear and allows reruns from intermediate checkpoints.

12. The repository is also organized as a data product. data/raw/ is source-controlled only as a location placeholder, data/corpus/ and data/processed/ hold generated intermediate corpora, figures/ stores the derived visualizations, and logs/ records per-stage runtime output and diagnostics. The tracked CSVs and PDFs in outputs/ and figures/ represent the final analytical surface of the project.

13. The key external dependencies materially shaping the architecture are Python for preprocessing/analysis, R with stylo for Delta stylometry, and CLTK for Latin linguistic processing. The Latin BERT model is the only substantive ML component, but it is compartmentalized so it cannot influence the classical stylometry pipeline.

14. In diagram terms, the important data flows are: Perseus XML in → cleaned corpus units → normalized/lemmatized corpora → feature matrices and stylo exports → parallel analysis branches → consolidated results/report. The most important boundaries are raw vs generated data, Python vs R analysis, and stylistic-feature methods vs semantic cross-checks.