# suhaib1769/thesis architecture

> Architecture diagram of the GitHub repository suhaib1769/thesis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/suhaib1769/thesis
- Repository: https://github.com/suhaib1769/thesis

## Overview

##### Project type and purpose
- This is a research-oriented data processing and neural information retrieval (IR) project (“Thesis – Exploring Neural Information in Europeana”). It ingests Europeana XML data and CLEF-CHIC topic/qrels files, processes them into indexable formats, runs various neural retrieval experiments (ColBERT, Hybrid, SPLADE, SBERT), and evaluates results against relevance judgments.

##### File structure
- Top-level directories:
- CHIC: CLEF-CHIC 2013 topic definitions and qrels in multiple languages.
- EDP: “Europeana Documents” raw XML data, data-collection and parsing scripts, pipeline notebooks, test files.
- NIR: Neural IR experiments subdivided into subfolders for different methods (ColBERT, Hybrid/Milvus, SPLADE, SBERT) containing notebooks and helper scripts.
- Tutorials: external tutorial materials (not part of core pipeline).
- myenv: Python virtual environment containing dependencies (rdflib, pyparsing, etc.).
- Patterns:
- Data‐ingestion layer (EDP/parser_*.py, data_collection).
- Batch pipeline orchestrated via notebooks and scripts, producing Solr XML outputs.
- Multiple IR modules connecting to storage/index layer (Solr and Milvus).
- Evaluation layer reading CHIC topics/qrels and retrieval outputs.

##### README for additional insights
- Very brief (“Thesis – Exploring Neural Information in Europeana”).
- Implicit technical stack: Python, Jupyter, Solr, Milvus, rdflib.

##### Elements
- Labels: Data Source, Ingestion Scripts, Parser, Solr Index, Milvus Index, Retrieval Modules (ColBERT / Hybrid / SPLADE / SBERT), Evaluation.
- From raw XML & topics/qrels → Ingestion & Parser
- Parser → Solr XML output → Solr
- Parser → Embedding generator → Milvus (for Hybrid)
- From Solr & Milvus → Retrieval modules → result files
- Result files & qrels → Evaluation metrics
- Blue rectangles for data sources
- Green for processing components
- Orange for storage layers
- Purple for retrieval engines
- Red for evaluation

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources
    subgraph "Data Sources"
        style DS fill:#bbe1fa,stroke:#0f4c75,stroke-width:2px
        RawXML["Europeana Raw XML\n<2021672>"]:::datasrc
        CHICTopics["CHIC Topics Definitions"]:::datasrc
        CHICQrelsPR["CHIC Qrels (with partially relevant)"]:::datasrc
        CHICQrelsML["CLEF-CHIC Qrels\n(multi-language)"]:::datasrc
    end

    %% Processing Layer
    subgraph "Data Ingestion & Processing"
        style DP fill:#d3f8e2,stroke:#1b4332,stroke-width:2px
        DataCollection["Data Collection Scripts & Notebooks"]:::process
        RDFConfig["RDF Schema & Config"]:::process
        Parser1["Parser Module\nparser_1.py"]:::process
        Parser2["Parser Module\nparser_2.py"]:::process
        ParserV1["Parser Module\nparser_v1.py"]:::process
        ParserTest["Parser Module\nparser2test.py"]:::process
        ParserNB["Parser Notebook"]:::process
    end

    %% Storage Layer
    subgraph "Index / Storage"
        style ST fill:#ffe5d9,stroke:#b22222,stroke-width:2px
        SolrXML["Solr XML Outputs"]:::storage
        SolrIndex["Solr Index"]:::storage
        MilvusDB["Milvus Vector Index"]:::storage
    end

    %% Retrieval Modules
    subgraph "Retrieval Modules"
        style RM fill:#e0c3fc,stroke:#5a189a,stroke-width:2px
        ColBERT["ColBERT Module"]:::retrieval
        Hybrid["Hybrid Module\n(Dense+Sparse)"]:::retrieval
        SPLADE["SPLADE Module"]:::retrieval
        SBERT["SBERT Module"]:::retrieval
    end

    %% Evaluation Layer
    subgraph "Evaluation"
        style EV fill:#ffccd5,stroke:#c9184a,stroke-width:2px
        Results["Retrieval Results Files"]:::evaluation
        EvalScripts["Evaluation Scripts & Notebooks"]:::evaluation
    end

    %% Environment
    myenv["Python Virtual Environment"]:::env

    %% Connections
    RawXML -->|XML input| DataCollection
    CHICTopics -->|Topics| EvalScripts
    CHICQrelsPR -->|Qrels| EvalScripts
    CHICQrelsML -->|Qrels| EvalScripts

    DataCollection -->|generates| RDFConfig
    DataCollection -->|feeds| Parser1
    DataCollection -->|feeds| ParserV1
    DataCollection -->|feeds| Parser2
    DataCollection -->|feeds| ParserTest
    DataCollection -->|feeds| ParserNB

    RDFConfig -->|config for| Parser1
    Parser1 -->|produces Solr XML| SolrXML
    ParserV1 -->|produces Solr XML| SolrXML
    Parser2 -->|produces embedding vectors| MilvusDB
    ParserNB -->|interactive parsing| SolrXML

    SolrXML -->|import XML| SolrIndex
    MilvusDB -->|load vectors| Hybrid

    SolrIndex -->|HTTP/Solr query| ColBERT
    SolrIndex -->|HTTP/Solr query| Hybrid
    SolrIndex -->|HTTP/Solr query| SPLADE
    SolrIndex -->|HTTP/Solr query| SBERT

    Hybrid -->|Milvus Python client| MilvusDB

    ColBERT -->|ranked lists| Results
    Hybrid -->|ranked lists| Results
    SPLADE -->|ranked lists| Results
    SBERT -->|ranked lists| Results

    Results -->|input| EvalScripts

    myenv --> DataCollection
    myenv --> Parser1
    myenv --> Parser2
    myenv --> ParserV1
    myenv --> ParserTest
    myenv --> ParserNB
    myenv --> ColBERT
    myenv --> Hybrid
    myenv --> SPLADE
    myenv --> SBERT
    myenv --> EvalScripts

    %% Click Events
    click CHICTopics "https://github.com/suhaib1769/thesis/tree/main/CHIC/CHIC-2013-AH-Topics"
    click CHICQrelsPR "https://github.com/suhaib1769/thesis/tree/main/CHIC/CHIC2013-AHqrels-withPartiallyRelevant"
    click CHICQrelsML "https://github.com/suhaib1769/thesis/tree/main/CHIC/CHiC2013-qrels"
    click RawXML "https://github.com/suhaib1769/thesis/tree/main/EDP/2021672"
    click RDFConfig "https://github.com/suhaib1769/thesis/blob/main/EDP/Europeana documents/schema.xml"
    click RDFConfig "https://github.com/suhaib1769/thesis/blob/main/EDP/Europeana documents/enumsConfig.xml"
    click DataCollection "https://github.com/suhaib1769/thesis/tree/main/EDP/data_collection"
    click Parser1 "https://github.com/suhaib1769/thesis/blob/main/EDP/parser_1.py"
    click Parser2 "https://github.com/suhaib1769/thesis/blob/main/EDP/parser_2.py"
    click ParserV1 "https://github.com/suhaib1769/thesis/blob/main/EDP/parser_v1.py"
    click ParserTest "https://github.com/suhaib1769/thesis/blob/main/EDP/parser2test.py"
    click ParserNB "https://github.com/suhaib1769/thesis/blob/main/EDP/parser.ipynb"
    click SolrXML "https://github.com/suhaib1769/thesis/blob/main/EDP/output_solr.xml"
    click SolrXML "https://github.com/suhaib1769/thesis/blob/main/EDP/output_solr2.xml"
    click SolrXML "https://github.com/suhaib1769/thesis/blob/main/EDP/output_solr3.xml"
    click SolrXML "https://github.com/suhaib1769/thesis/blob/main/EDP/output_solr33.xml"
    click SolrXML "https://github.com/suhaib1769/thesis/blob/main/EDP/output_solr34.xml"
    click ColBERT "https://github.com/suhaib1769/thesis/tree/main/NIR/Colbert"
    click Hybrid "https://github.com/suhaib1769/thesis/tree/main/NIR/Hybrid"
    click SPLADE "https://github.com/suhaib1769/thesis/tree/main/NIR/SPLADE_test"
    click SBERT "https://github.com/suhaib1769/thesis/tree/main/NIR/sbert"
    click myenv "https://github.com/suhaib1769/thesis/tree/main/myenv/"

    %% Styles
    classDef datasrc fill:#bbe1fa,stroke:#0f4c75,color:#0f4c75;
    classDef process fill:#d3f8e2,stroke:#1b4332,color:#1b4332;
    classDef storage fill:#ffe5d9,stroke:#b22222,color:#b22222;
    classDef retrieval fill:#e0c3fc,stroke:#5a189a,color:#5a189a;
    classDef evaluation fill:#ffccd5,stroke:#c9184a,color:#c9184a;
    classDef env fill:#e0e0e0,stroke:#666666,color:#333333;
```
