# bfiliks/extractvariants architecture

> Architecture diagram of the GitHub repository bfiliks/extractvariants, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bfiliks/extractvariants
- Repository: https://github.com/bfiliks/extractvariants

## Overview

##### Project Type and Purpose
- This is a data‐visualization/ETL pipeline project built in Python (plus an XQuery script).
- Purpose: scrape Emily Dickinson Archive (EDA) TEI XML files (no public API), extract and transform variant data into CSV, then produce interactive and static visualizations (Plotly, Matplotlib, etc.).

##### File‐Structure Analysis
- Top level:
- README.md, LICENSE.md, .gitignore
- code/ (all extraction & graph‐generation scripts + code_base.md)
- data/ (raw TEI HTML, intermediate CSVs)
- data_visualization/ (final images/HTML embeds)
- Patterns: single‐project ETL → visualization pipeline. No separate “frontend” or “backend” services—everything runs locally or in notebooks.
- Key scripts:
- tei_data_extraction.py – scrapes/parses TEI XML → dataxml.csv
- eda_counting_robot.xq – XQuery variant counting over TEI
- *_graph.py – Python scripts to transform CSVs → interactive and static graphs
- Data folders hold both raw inputs and transformed outputs.

##### README Insights
- Describes lack of API → scraping approach.
- Emphasizes data management plan (formats, version control).
- Visuals embedded via Plotly iframes and other outputs in data_visualization/.

a. Main Components
1. External Data Source
– Emily Dickinson Archive website (TEI‐encoded documents)
2. Extraction Layer
– tei_data_extraction.py (Python scraper + XML parser)
– eda_counting_robot.xq (XQuery processor)
3. Data Storage
– Raw files (tei_raw_files.html)
– Intermediate CSVs (dataxml.csv, basevariantsbycount.csv, modified_eda.csv)
4. Processing/Transformation Layer
– Python scripts:
- year_range_interactive_graph.py
- fascicle_range_interactive_graph.py
- weighted_network_graph.py
– Data cleaning, aggregation, network computations
5. Visualization Layer
– Output folder data_visualization/
– Plotly embeddings + static image exports
6. Documentation
– code/code_base.md
– README.md
– Data management planning steps

b. Relationships & Data Flow
- Extraction Layer → Raw Storage (tei_raw_files.html)
- Extraction Layer → Intermediate CSV Storage
- Processing Layer reads CSVs → produces visualization artifacts
- Artifacts written to data_visualization/
- Documentation sits alongside code to guide future runs/maintenance

c. Architectural Patterns & Principles
– Clear separation of concerns (Extraction vs. Transformation vs. Visualization)
– File‐system–based “data lake” for raw & processed data
– Minimal dependencies: Python + lxml/XML/XQuery + Plotly/Matplotlib
– Batch‐style ETL (no real‐time component)

d. Key Technologies
– Python (Scrapy or requests + BeautifulSoup/lxml)
– XQuery (eda_counting_robot.xq)
– CSV (pandas)
– Plotly for interactive charts
– matplotlib or seaborn for static plots

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Source
    ext["Emily Dickinson Archive Website"]:::external

    %% Extraction Layer
    subgraph "Extraction Layer"
        direction TB
        scrape["tei_data_extraction.py"]:::extraction
        xquery["eda_counting_robot.xq"]:::extraction
    end

    %% Data Storage
    subgraph "Data Storage"
        direction TB
        raw["tei_raw_files.html"]:::storage
        csv1["dataxml.csv"]:::storage
        csv2["basevariantsbycount.csv"]:::storage
        csv3["modified_eda.csv"]:::storage
    end

    %% Processing/Transformation Layer
    subgraph "Processing/Transformation Layer"
        direction TB
        pr1["year_range_interactive_graph.py"]:::transformation
        pr2["fascicle_range_interactive_graph.py"]:::transformation
        pr3["weighted_network_graph.py"]:::transformation
    end

    %% Visualization Layer
    subgraph "Visualization Layer"
        direction TB
        viz["data_visualization/"]:::visualization
    end

    %% Documentation
    subgraph "Documentation"
        direction TB
        doc1["code/code_base.md"]:::documentation
        doc2["README.md"]:::documentation
    end

    %% Data Flow
    ext --> scrape
    ext --> xquery

    scrape --> raw
    scrape --> csv1
    xquery --> csv2
    xquery --> csv3

    csv1 --> pr1
    csv1 --> pr2
    csv2 --> pr3
    csv3 --> pr3

    pr1 --> viz
    pr2 --> viz
    pr3 --> viz

    %% Click Events
    click scrape "https://github.com/bfiliks/extractvariants/blob/main/code/tei_data_extraction.py"
    click xquery "https://github.com/bfiliks/extractvariants/blob/main/code/eda_counting_robot.xq"
    click raw "https://github.com/bfiliks/extractvariants/blob/main/data/tei_raw_files.html"
    click csv1 "https://github.com/bfiliks/extractvariants/blob/main/data/dataxml.csv"
    click csv2 "https://github.com/bfiliks/extractvariants/blob/main/data/basevariantsbycount.csv"
    click csv3 "https://github.com/bfiliks/extractvariants/blob/main/data/modified_eda.csv"
    click pr1 "https://github.com/bfiliks/extractvariants/blob/main/code/year_range_interactive_graph.py"
    click pr2 "https://github.com/bfiliks/extractvariants/blob/main/code/fascicle_range_interactive_graph.py"
    click pr3 "https://github.com/bfiliks/extractvariants/blob/main/code/weighted_network_graph.py"
    click viz "https://github.com/bfiliks/extractvariants/tree/main/data_visualization/"
    click doc1 "https://github.com/bfiliks/extractvariants/blob/main/code/code_base.md"
    click doc2 "https://github.com/bfiliks/extractvariants/blob/main/README.md"

    %% Styles
    classDef extraction fill:#cce5ff,stroke:#005cbf,color:#005cbf
    classDef storage fill:#d4edda,stroke:#155724,color:#155724
    classDef transformation fill:#ffe5b4,stroke:#ff8c00,color:#ff8c00
    classDef visualization fill:#e2d6f9,stroke:#6f42c1,color:#6f42c1
    classDef documentation fill:#f0f0f0,stroke:#6c757d,color:#6c757d
    classDef external fill:#f0f8ff,stroke:#17a2b8,color:#17a2b8
```
