# durovo51/wikipedia-mock-search-engine architecture

> Architecture diagram of the GitHub repository durovo51/wikipedia-mock-search-engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/durovo51/wikipedia-mock-search-engine
- Repository: https://github.com/durovo51/wikipedia-mock-search-engine

## Overview

WikiSearch ingests Wikipedia XML into a cleaned SQLite corpus, tokenizes documents to build a persisted inverted index, and serves user queries through Streamlit. The principal search path tokenizes a query, finds candidate documents, scores them with custom BM25, then fetches and displays document details. The UI also offers a separate naive substring-search comparison. The graph follows the sampled source; the README documents parsing and ranking capabilities, but no additional implementation paths are inferred.

Main entry points: [`streamlit_frontend.py`](https://github.com/durovo51/wikipedia-mock-search-engine/blob/main/streamlit_frontend.py) (Streamlit interface), [`searching.py`](https://github.com/durovo51/wikipedia-mock-search-engine/blob/main/searching.py) (Query search), [`inverted_index.py`](https://github.com/durovo51/wikipedia-mock-search-engine/blob/main/inverted_index.py) (Index builder), [`wiki_corpus.db`](https://github.com/durovo51/wikipedia-mock-search-engine/blob/main/wiki_corpus.db) (SQLite corpus)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Corpus ingestion"]
  node_ingest["Parse and clean"]
  node_corpus[("SQLite corpus<br/>[wiki_corpus.db]")]
end

subgraph group_indexing["Index construction"]
  node_tokenizer["Text tokenizer"]
  node_builder["Index builder<br/>[inverted_index.py]"]
  node_index["Inverted index<br/>[inverted_index.pkl]"]
  node_lengths["Document lengths<br/>[doc_lengths.pkl]"]
end

subgraph group_runtime["Search runtime"]
  node_search["Query search<br/>[searching.py]"]
  node_ranking["BM25 ranking<br/>[ranking.py]"]
end

subgraph group_interface["User interface"]
  node_ui["Streamlit interface"]
  node_results["Displayed results"]
end

node_wiki_dump["Wikipedia dump"]
node_user(("Search user"))

node_wiki_dump -->|"parses"| node_ingest
node_ingest -->|"writes documents"| node_corpus
node_builder -->|"reads documents"| node_corpus
node_builder -->|"tokenizes text"| node_tokenizer
node_builder -->|"writes term counts"| node_index
node_builder -->|"writes lengths"| node_lengths
node_user -->|"enters query"| node_ui
node_ui -->|"runs query"| node_search
node_search -->|"tokenizes query"| node_tokenizer
node_search -->|"looks up terms"| node_index
node_search -->|"scores candidates"| node_ranking
node_ranking -->|"reads term counts"| node_index
node_ranking -->|"reads lengths"| node_lengths
node_search -->|"returns ranked IDs"| node_ui
node_ui -->|"fetches document details"| node_corpus
node_ui -->|"displays results"| node_results
node_ui -->|"runs substring query"| node_corpus

click node_ingest "https://github.com/durovo51/wikipedia-mock-search-engine/blob/main/clean_up_and_connect_to_db.py"
click node_corpus "https://github.com/durovo51/wikipedia-mock-search-engine/blob/main/wiki_corpus.db"
click node_tokenizer "https://github.com/durovo51/wikipedia-mock-search-engine/blob/main/tokenize_and_stem_pipeline.py"
click node_builder "https://github.com/durovo51/wikipedia-mock-search-engine/blob/main/inverted_index.py"
click node_index "https://github.com/durovo51/wikipedia-mock-search-engine/blob/main/inverted_index.pkl"
click node_lengths "https://github.com/durovo51/wikipedia-mock-search-engine/blob/main/doc_lengths.pkl"
click node_ui "https://github.com/durovo51/wikipedia-mock-search-engine/blob/main/streamlit_frontend.py"
click node_search "https://github.com/durovo51/wikipedia-mock-search-engine/blob/main/searching.py"
click node_ranking "https://github.com/durovo51/wikipedia-mock-search-engine/blob/main/ranking.py"
click node_results "https://github.com/durovo51/wikipedia-mock-search-engine/blob/main/streamlit_frontend.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ingest,node_corpus,node_user toneBlue
class node_tokenizer,node_builder,node_index,node_lengths toneAmber
class node_search,node_ranking toneMint
class node_ui,node_results toneRose
class node_wiki_dump toneIndigo
```

## Components

### Corpus ingestion

- [Parse and clean](https://github.com/durovo51/wikipedia-mock-search-engine/blob/main/clean_up_and_connect_to_db.py): `clean_up_and_connect_to_db.py`
- [SQLite corpus](https://github.com/durovo51/wikipedia-mock-search-engine/blob/main/wiki_corpus.db): `wiki_corpus.db`

### Index construction

- [Text tokenizer](https://github.com/durovo51/wikipedia-mock-search-engine/blob/main/tokenize_and_stem_pipeline.py): `tokenize_and_stem_pipeline.py`
- [Index builder](https://github.com/durovo51/wikipedia-mock-search-engine/blob/main/inverted_index.py): `inverted_index.py`
- [Inverted index](https://github.com/durovo51/wikipedia-mock-search-engine/blob/main/inverted_index.pkl): `inverted_index.pkl`
- [Document lengths](https://github.com/durovo51/wikipedia-mock-search-engine/blob/main/doc_lengths.pkl): `doc_lengths.pkl`

### Search runtime

- [Query search](https://github.com/durovo51/wikipedia-mock-search-engine/blob/main/searching.py): `searching.py`
- [BM25 ranking](https://github.com/durovo51/wikipedia-mock-search-engine/blob/main/ranking.py): `ranking.py`

### User interface

- [Streamlit interface](https://github.com/durovo51/wikipedia-mock-search-engine/blob/main/streamlit_frontend.py): `streamlit_frontend.py`
- [Displayed results](https://github.com/durovo51/wikipedia-mock-search-engine/blob/main/streamlit_frontend.py): `streamlit_frontend.py`

### Other components

- **Wikipedia dump**
- **Search user**

## Connections

- Wikipedia dump → Parse and clean: parses
- Parse and clean → SQLite corpus: writes documents
- Index builder → SQLite corpus: reads documents
- Index builder → Text tokenizer: tokenizes text
- Index builder → Inverted index: writes term counts
- Index builder → Document lengths: writes lengths
- Search user → Streamlit interface: enters query
- Streamlit interface → Query search: runs query
- Query search → Text tokenizer: tokenizes query
- Query search → Inverted index: looks up terms
- Query search → BM25 ranking: scores candidates
- BM25 ranking → Inverted index: reads term counts
- BM25 ranking → Document lengths: reads lengths
- Query search → Streamlit interface: returns ranked IDs
- Streamlit interface → SQLite corpus: fetches document details
- Streamlit interface → Displayed results: displays results
- Streamlit interface → SQLite corpus: runs substring query