# aubinx/search-engine-headmind-partners architecture

> Architecture diagram of the GitHub repository aubinx/search-engine-headmind-partners, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aubinx/search-engine-headmind-partners
- Repository: https://github.com/aubinx/search-engine-headmind-partners

## Overview

##### Main Components
- Data Ingestion & Preprocessing:
- Ingestion of the “datascience.stackexchange.com” folder (as required by the README) which serves as the raw document source.
- Preprocessing to generate the inverted index (both content and tags) and other pickled data files (“inverted_index.pickle”, “inverted_index_tags.pkl”, “posts.pkl”, “tag_to_processed_tag_dict.pickle”, “vi_sums.pkl”). This module prepares and structures the data for fast retrieval.
- Core Search Engine Logic:
- The central function “CobraSearch” that, given a query, computes scores for documents.
- The scoring process that uses cosine similarity for content, and additional weights from votes, comments, user badges, and title relevance.
- Any intermediary steps that process the query to hit the preprocessed indices.
- User Interface:
- The tkinter-based GUI provided towards the end of the Search_engine_final notebook. This component captures user input and displays ranked search results.
- Experimentation & Evaluation:
- The separate Jupyter Notebook files (Day1, Day2, Day3) that may represent exploratory data analysis, algorithm tuning, or evaluation of ranking metrics.
- Supporting Modules & Dependencies:
- The requirements.txt file indicating the Python libraries and dependencies.
- Documentation files (README.md, Rapport_EI_Groupe4.docx, evaluation Excel file) that help understand usage and evaluation benchmarks.

##### Relationships and Interactions
- Data flows from the raw data source (the unzipped “datascience.stackexchange.com” folder) into the preprocessing component.
- Preprocessed data (inverted indices and other pickled data) is then loaded by the search engine as part of query processing.
- When a user inputs a query (via the tkinter GUI or directly via notebook interface), the “CobraSearch” function orchestrates:
- Query parsing
- Retrieval of relevant documents using the inverted indices
- Applying cosine similarity and further weighting calculations (votes, badges, comments)
- Returning search results.
- The output is then displayed either through the GUI or in the notebook output.

##### Architectural Patterns and Design Principles
- Modularity: Separate functionalities (data ingestion, query processing, and UI) ensure easy maintenance and potential extension.
- Preprocessing & Caching: Using pickled data structures for the inverted index and other mappings emphasizes the principle of computing expensive operations up-front to ensure fast run-time performance.
- Experimentation: The presence of multiple notebook files indicates an iterative approach to refining algorithms and evaluation metrics.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Raw Data
    RawData["Raw Data: datascience.stackexchange.com"]:::raw

    %% Data Ingestion & Preprocessing
    DataPreproc["Data Preprocessing Module"]:::processing

    %% Preprocessed Data Files Subgraph
    subgraph "Preprocessed Data Files"
        File1["inverted_index.pickle"]:::support
        File2["inverted_index_tags.pkl"]:::support
        File3["posts.pkl"]:::support
        File4["tag_to_processed_tag_dict.pickle"]:::support
        File5["vi_sums.pkl"]:::support
    end

    %% Core Search Engine Logic Subgraph
    subgraph "Core Search Engine"
        Engine["CobraSearch (Search_engine_final.ipynb)"]:::engine
        Results["Search Results"]:::engine
    end

    %% User Interface Subgraph
    subgraph "User Interface"
        UI["tkinter GUI"]:::ui
    end

    %% Experimentation & Evaluation Subgraph
    subgraph "Experimentation & Evaluation"
        Exp1["Day1.ipynb"]:::experiment
        Exp2["Day2.ipynb"]:::experiment
        Exp3["Day3.ipynb"]:::experiment
    end

    %% Supporting Modules & Dependencies Subgraph
    subgraph "Supporting Modules & Dependencies"
        Dep["requirements.txt"]:::support
        Doc1["README.md"]:::support
        Doc2["Rapport_EI_Groupe4.docx"]:::support
        Doc3["evaluation_search_engine_post_queries_ranking_EI_CS.xlsx"]:::support
    end

    %% Connections
    RawData -->|"feeds"| DataPreproc
    DataPreproc --> File1
    DataPreproc --> File2
    DataPreproc --> File3
    DataPreproc --> File4
    DataPreproc --> File5
    File1 -->|"loads"| Engine
    File2 -->|"loads"| Engine
    File3 -->|"loads"| Engine
    File4 -->|"loads"| Engine
    File5 -->|"loads"| Engine
    UI -->|"sendsQuery"| Engine
    Engine -->|"returns"| Results

    %% Class Definitions for Colors
    classDef raw fill:#AED6F1,stroke:#1B4F72,stroke-width:2px;
    classDef processing fill:#A9DFBF,stroke:#1D8348,stroke-width:2px;
    classDef engine fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef ui fill:#FAD7A0,stroke:#D35400,stroke-width:2px;
    classDef experiment fill:#F5B7B1,stroke:#922B21,stroke-width:2px;
    classDef support fill:#D5DBDB,stroke:#566573,stroke-width:2px;

    %% Click Events
    click File1 "https://github.com/aubinx/search-engine-headmind-partners/blob/main/inverted_index.pickle"
    click File2 "https://github.com/aubinx/search-engine-headmind-partners/blob/main/inverted_index_tags.pkl"
    click File3 "https://github.com/aubinx/search-engine-headmind-partners/blob/main/posts.pkl"
    click File4 "https://github.com/aubinx/search-engine-headmind-partners/blob/main/tag_to_processed_tag_dict.pickle"
    click File5 "https://github.com/aubinx/search-engine-headmind-partners/blob/main/vi_sums.pkl"
    click Engine "https://github.com/aubinx/search-engine-headmind-partners/blob/main/Search_engine_final.ipynb"
    click Exp1 "https://github.com/aubinx/search-engine-headmind-partners/blob/main/Day1.ipynb"
    click Exp2 "https://github.com/aubinx/search-engine-headmind-partners/blob/main/Day2.ipynb"
    click Exp3 "https://github.com/aubinx/search-engine-headmind-partners/blob/main/Day3.ipynb"
    click Dep "https://github.com/aubinx/search-engine-headmind-partners/blob/main/requirements.txt"
    click Doc1 "https://github.com/aubinx/search-engine-headmind-partners/blob/main/README.md"
    click Doc2 "https://github.com/aubinx/search-engine-headmind-partners/blob/main/Rapport_EI_Groupe4.docx"
    click Doc3 "https://github.com/aubinx/search-engine-headmind-partners/blob/main/evaluation_search_engine_post_queries_ranking_EI_CS.xlsx"
```
