How aubinx/search-engine-headmind-partners works
Main Components
• Data Ingestion & Preprocessing:
• Ingestion of the “datascience.stackexchange.com” folder (as required by the README) which serves as the raw document source.
• Preprocessing to generate the inverted index (both content and tags) and other pickled data files (“inverted_index.pickle”, “inverted_index_tags.pkl”, “posts.pkl”, “tag_to_processed_tag_dict.pickle”, “vi_sums.pkl”). This module prepares and structures the data for fast retrieval.
• Core Search Engine Logic:
• The central function “CobraSearch” that, given a query, computes scores for documents.
• The scoring process that uses cosine similarity for content, and additional weights from votes, comments, user badges, and title relevance.
• Any intermediary steps that process the query to hit the preprocessed indices.
• User Interface:
• The tkinter-based GUI provided towards the end of the Search_engine_final notebook. This component captures user input and displays ranked search results.
• Experimentation & Evaluation:
• The separate Jupyter Notebook files (Day1, Day2, Day3) that may represent exploratory data analysis, algorithm tuning, or evaluation of ranking metrics.
• Supporting Modules & Dependencies:
• The requirements.txt file indicating the Python libraries and dependencies.
• Documentation files (README.md, Rapport_EI_Groupe4.docx, evaluation Excel file) that help understand usage and evaluation benchmarks.
Relationships and Interactions
• Data flows from the raw data source (the unzipped “datascience.stackexchange.com” folder) into the preprocessing component.
• Preprocessed data (inverted indices and other pickled data) is then loaded by the search engine as part of query processing.
• When a user inputs a query (via the tkinter GUI or directly via notebook interface), the “CobraSearch” function orchestrates:
• Query parsing
• Retrieval of relevant documents using the inverted indices
• Applying cosine similarity and further weighting calculations (votes, badges, comments)
• Returning search results.
• The output is then displayed either through the GUI or in the notebook output.
Architectural Patterns and Design Principles
• Modularity: Separate functionalities (data ingestion, query processing, and UI) ensure easy maintenance and potential extension.
• Preprocessing & Caching: Using pickled data structures for the inverted index and other mappings emphasizes the principle of computing expensive operations up-front to ensure fast run-time performance.
• Experimentation: The presence of multiple notebook files indicates an iterative approach to refining algorithms and evaluation metrics.