How suhaib1769/thesis works
Project type and purpose
• This is a research-oriented data processing and neural information retrieval (IR) project (“Thesis – Exploring Neural Information in Europeana”). It ingests Europeana XML data and CLEF-CHIC topic/qrels files, processes them into indexable formats, runs various neural retrieval experiments (ColBERT, Hybrid, SPLADE, SBERT), and evaluates results against relevance judgments.
File structure
• Top-level directories:
• CHIC: CLEF-CHIC 2013 topic definitions and qrels in multiple languages.
• EDP: “Europeana Documents” raw XML data, data-collection and parsing scripts, pipeline notebooks, test files.
• NIR: Neural IR experiments subdivided into subfolders for different methods (ColBERT, Hybrid/Milvus, SPLADE, SBERT) containing notebooks and helper scripts.
• Tutorials: external tutorial materials (not part of core pipeline).
• myenv: Python virtual environment containing dependencies (rdflib, pyparsing, etc.).
• Patterns:
• Data‐ingestion layer (EDP/parser_*.py, data_collection).
• Batch pipeline orchestrated via notebooks and scripts, producing Solr XML outputs.
• Multiple IR modules connecting to storage/index layer (Solr and Milvus).
• Evaluation layer reading CHIC topics/qrels and retrieval outputs.
README for additional insights
• Very brief (“Thesis – Exploring Neural Information in Europeana”).
• Implicit technical stack: Python, Jupyter, Solr, Milvus, rdflib.
Elements
• Labels: Data Source, Ingestion Scripts, Parser, Solr Index, Milvus Index, Retrieval Modules (ColBERT / Hybrid / SPLADE / SBERT), Evaluation.
• From raw XML & topics/qrels → Ingestion & Parser
• Parser → Solr XML output → Solr
• Parser → Embedding generator → Milvus (for Hybrid)
• From Solr & Milvus → Retrieval modules → result files
• Result files & qrels → Evaluation metrics
• Blue rectangles for data sources
• Green for processing components
• Orange for storage layers
• Purple for retrieval engines
• Red for evaluation