How sharvaridhote/wikipedia_citation_project works
Project type and purpose
• This is a machine‐learning pipeline / research project for detecting “Citation Needed” statements in Wikipedia text.
• The goal is to scrape raw Wikipedia content, clean and tokenize sentences, train a classification model (using spaCy or similar), store model artifacts, and expose an inference interface (scripts/app.py).
File structure
Top‐level directories:
• data/ – raw and processed datasets (CSV, TXT).
• models/ – multiple trained spaCy model artifacts (NER, parser, tagger, textcat, tokenizer, vocab).
• notebooks/ – exploratory data analysis and hyperparameter tuning experiments.
• scripts/ – the core Python components:
– scraper.py (collects raw text),
– sentencetokenizer.py & util.py (data preprocessing),
– train.py (model training pipeline),
– app.py (model serving/inference API),
– main.py (orchestration entry point).
Supporting files: .gitignore, LICENSE, README.md.
README
• Minimal description: “Wikipedia Citation Needed Model.”
a. Main components
1. Data Sources: external Wikipedia dumps or online pages
2. Scraper Module (scripts/scraper.py)
3. Raw Data Storage (data/raw/*)
4. Preprocessor Module (scripts/sentencetokenizer.py, util.py)
5. Processed Data Storage (data/processed/*.csv)
6. Experimentation Environment (notebooks/)
7. Training Module (scripts/train.py)
8. Model Registry / Artifact Store (models/)
9. Inference API (scripts/app.py)
10. Orchestrator (scripts/main.py)
b. Relationships and data flow
• Scraper → writes raw files to data/raw → Preprocessor reads raw → writes data/processed
• Notebooks read data/processed for EDA and hyperparameter tuning (side‐channel for human analysis)
• Training Module reads data/processed → trains model → writes artifacts under models/
• Inference API loads model from models/ → serves predictions via REST or CLI
• main.py glues steps in sequence or schedules pipelines
c. Architectural patterns & design principles
• Modular pipeline: each step isolated in its own script
• Clear separation between data ingestion, processing, training, and serving
• Artifact versioning: each model labeled by hyperparameters in its folder
• Experimentation vs. production code separated (notebooks vs. scripts)
d. Key technologies and libraries
• Python scripts (scraper, tokenizer, train, app)
• spaCy for NLP pipelines (NER, parser, textcat)
• pandas/numpy for data processing
• Jupyter notebooks for EDA & experiments
• (Optional) Flask or FastAPI in app.py for serving
Detailed capture of essential elements
• Data ingestion: scraper.py pulling from external Wikipedia
• Data transformation: sentencetokenizer.py and util functions producing cleaned CSVs
• Experiment branch: notebooks for EDA and hyperparameter tuning
• Training pipeline: train.py orchestrating data loading, spaCy pipeline setup, model.fit
• Model artifact layout: metadata, tokenizer, parser, tagger, textcat subfolders
• Serving layer: app.py exposing endpoints for text input → “citation needed?” prediction
• Orchestrator/main.py: script to run end‐to‐end workflow or schedule jobs