How bfiliks/extractvariants works
Project Type and Purpose
• This is a data‐visualization/ETL pipeline project built in Python (plus an XQuery script).
• Purpose: scrape Emily Dickinson Archive (EDA) TEI XML files (no public API), extract and transform variant data into CSV, then produce interactive and static visualizations (Plotly, Matplotlib, etc.).
File‐Structure Analysis
• Top level:
• README.md, LICENSE.md, .gitignore
• code/ (all extraction & graph‐generation scripts + code_base.md)
• data/ (raw TEI HTML, intermediate CSVs)
• data_visualization/ (final images/HTML embeds)
• Patterns: single‐project ETL → visualization pipeline. No separate “frontend” or “backend” services—everything runs locally or in notebooks.
• Key scripts:
• tei_data_extraction.py – scrapes/parses TEI XML → dataxml.csv
• eda_counting_robot.xq – XQuery variant counting over TEI
• *_graph.py – Python scripts to transform CSVs → interactive and static graphs
• Data folders hold both raw inputs and transformed outputs.
README Insights
• Describes lack of API → scraping approach.
• Emphasizes data management plan (formats, version control).
• Visuals embedded via Plotly iframes and other outputs in data_visualization/.
a. Main Components
1. External Data Source
– Emily Dickinson Archive website (TEI‐encoded documents)
2. Extraction Layer
– tei_data_extraction.py (Python scraper + XML parser)
– eda_counting_robot.xq (XQuery processor)
3. Data Storage
– Raw files (tei_raw_files.html)
– Intermediate CSVs (dataxml.csv, basevariantsbycount.csv, modified_eda.csv)
4. Processing/Transformation Layer
– Python scripts:
• year_range_interactive_graph.py
• fascicle_range_interactive_graph.py
• weighted_network_graph.py
– Data cleaning, aggregation, network computations
5. Visualization Layer
– Output folder data_visualization/
– Plotly embeddings + static image exports
6. Documentation
– code/code_base.md
– README.md
– Data management planning steps
b. Relationships & Data Flow
• Extraction Layer → Raw Storage (tei_raw_files.html)
• Extraction Layer → Intermediate CSV Storage
• Processing Layer reads CSVs → produces visualization artifacts
• Artifacts written to data_visualization/
• Documentation sits alongside code to guide future runs/maintenance
c. Architectural Patterns & Principles
– Clear separation of concerns (Extraction vs. Transformation vs. Visualization)
– File‐system–based “data lake” for raw & processed data
– Minimal dependencies: Python + lxml/XML/XQuery + Plotly/Matplotlib
– Batch‐style ETL (no real‐time component)
d. Key Technologies
– Python (Scrapy or requests + BeautifulSoup/lxml)
– XQuery (eda_counting_robot.xq)
– CSV (pandas)
– Plotly for interactive charts
– matplotlib or seaborn for static plots