# sharvaridhote/wikipedia_citation_project architecture

> Architecture diagram of the GitHub repository sharvaridhote/wikipedia_citation_project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sharvaridhote/wikipedia_citation_project
- Repository: https://github.com/sharvaridhote/wikipedia_citation_project

## Overview

##### Project type and purpose
- This is a machine‐learning pipeline / research project for detecting “Citation Needed” statements in Wikipedia text.
- The goal is to scrape raw Wikipedia content, clean and tokenize sentences, train a classification model (using spaCy or similar), store model artifacts, and expose an inference interface (scripts/app.py).

##### File structure
Top‐level directories:
- data/ – raw and processed datasets (CSV, TXT).
- models/ – multiple trained spaCy model artifacts (NER, parser, tagger, textcat, tokenizer, vocab).
- notebooks/ – exploratory data analysis and hyperparameter tuning experiments.
- scripts/ – the core Python components:
– scraper.py (collects raw text),
– sentencetokenizer.py & util.py (data preprocessing),
– train.py (model training pipeline),
– app.py (model serving/inference API),
– main.py (orchestration entry point).

Supporting files: .gitignore, LICENSE, README.md.

##### README
- Minimal description: “Wikipedia Citation Needed Model.”

a. Main components
1. Data Sources: external Wikipedia dumps or online pages
2. Scraper Module (scripts/scraper.py)
3. Raw Data Storage (data/raw/*)
4. Preprocessor Module (scripts/sentencetokenizer.py, util.py)
5. Processed Data Storage (data/processed/*.csv)
6. Experimentation Environment (notebooks/)
7. Training Module (scripts/train.py)
8. Model Registry / Artifact Store (models/)
9. Inference API (scripts/app.py)
10. Orchestrator (scripts/main.py)

b. Relationships and data flow
- Scraper → writes raw files to data/raw → Preprocessor reads raw → writes data/processed
- Notebooks read data/processed for EDA and hyperparameter tuning (side‐channel for human analysis)
- Training Module reads data/processed → trains model → writes artifacts under models/
- Inference API loads model from models/ → serves predictions via REST or CLI
- main.py glues steps in sequence or schedules pipelines

c. Architectural patterns & design principles
- Modular pipeline: each step isolated in its own script
- Clear separation between data ingestion, processing, training, and serving
- Artifact versioning: each model labeled by hyperparameters in its folder
- Experimentation vs. production code separated (notebooks vs. scripts)

d. Key technologies and libraries
- Python scripts (scraper, tokenizer, train, app)
- spaCy for NLP pipelines (NER, parser, textcat)
- pandas/numpy for data processing
- Jupyter notebooks for EDA & experiments
- (Optional) Flask or FastAPI in app.py for serving

##### Detailed capture of essential elements
- Data ingestion: scraper.py pulling from external Wikipedia
- Data transformation: sentencetokenizer.py and util functions producing cleaned CSVs
- Experiment branch: notebooks for EDA and hyperparameter tuning
- Training pipeline: train.py orchestrating data loading, spaCy pipeline setup, model.fit
- Model artifact layout: metadata, tokenizer, parser, tagger, textcat subfolders
- Serving layer: app.py exposing endpoints for text input → “citation needed?” prediction
- Orchestrator/main.py: script to run end‐to‐end workflow or schedule jobs

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Source
    Wikipedia["Wikipedia Dumps"]:::external

    %% Orchestrator
    Orchestrator["Orchestrator / Pipeline Entry Point"]:::compute

    %% Data Ingestion
    Scraper["Scraper Module"]:::compute
    RawData[(Raw Data Storage)]:::data-store

    %% Preprocessing
    Preprocessor["Preprocessor Module<br>(Sentence Tokenizer)"]:::compute
    Util["Preprocessor Utility Functions"]:::compute
    ProcessedData[(Processed Data Storage)]:::data-store

    %% Experimentation
    Notebooks["Experimentation Environment"]:::notebook

    %% Training
    Trainer["Training Module"]:::compute
    ModelStore[(Model Registry / Artifact Store)]:::data-store

    %% Inference
    API["Inference API"]:::compute

    %% Documentation
    subgraph "Project Documentation"
        README["README.md"]:::docs
        Gitignore[".gitignore"]:::docs
        License["LICENSE"]:::docs
    end

    %% Flow
    Orchestrator -->|"runs pipeline"| Scraper
    Wikipedia -->|"raw pages"| Scraper
    Scraper -->|".txt files"| RawData
    RawData -->|"raw text"| Preprocessor
    Util -->|"helpers"| Preprocessor
    Preprocessor -->|".csv"| ProcessedData
    ProcessedData -->|".csv"| Notebooks
    Notebooks -->|"analysis insights"| Trainer
    ProcessedData -->|"training data"| Trainer
    Trainer -->|"model binaries"| ModelStore
    ModelStore -->|"load model"| API
    Orchestrator -->|"trigger retrain"| Trainer
    Orchestrator -->|"deploy service"| API

    %% Click Events
    click Scraper "https://github.com/sharvaridhote/wikipedia_citation_project/blob/main/scripts/scraper.py"
    click RawData "https://github.com/sharvaridhote/wikipedia_citation_project/tree/main/data/raw/"
    click Preprocessor "https://github.com/sharvaridhote/wikipedia_citation_project/blob/main/scripts/sentencetokenizer.py"
    click Util "https://github.com/sharvaridhote/wikipedia_citation_project/blob/main/scripts/util.py"
    click ProcessedData "https://github.com/sharvaridhote/wikipedia_citation_project/tree/main/data/processed/"
    click Notebooks "https://github.com/sharvaridhote/wikipedia_citation_project/tree/main/notebooks/"
    click Trainer "https://github.com/sharvaridhote/wikipedia_citation_project/blob/main/scripts/train.py"
    click ModelStore "https://github.com/sharvaridhote/wikipedia_citation_project/tree/main/models/"
    click API "https://github.com/sharvaridhote/wikipedia_citation_project/blob/main/scripts/app.py"
    click Orchestrator "https://github.com/sharvaridhote/wikipedia_citation_project/blob/main/scripts/main.py"
    click README "https://github.com/sharvaridhote/wikipedia_citation_project/blob/main/README.md"
    click Gitignore "https://github.com/sharvaridhote/wikipedia_citation_project/blob/main/.gitignore"
    click License "https://github.com/sharvaridhote/wikipedia_citation_project/tree/main/LICENSE"

    %% Styles
    classDef data-store fill:#c6f5c6,stroke:#2d7a2d,stroke-width:1px;
    classDef compute fill:#a3c4f3,stroke:#1f3e75,stroke-width:1px;
    classDef notebook fill:#f3e8a3,stroke:#7a6f1f,stroke-width:1px;
    classDef external fill:#f5c6a3,stroke:#7a3e1f,stroke-width:1px;
    classDef docs fill:#d3a3f3,stroke:#5d1f75,stroke-width:1px;
```
