How durovo51/wikipedia-mock-search-engine works
WikiSearch ingests Wikipedia XML into a cleaned SQLite corpus, tokenizes documents to build a persisted inverted index, and serves user queries through Streamlit. The principal search path tokenizes a query, finds candidate documents, scores them with custom BM25, then fetches and displays document details. The UI also offers a separate naive substring-search comparison. The graph follows the sampled source; the README documents parsing and ranking capabilities, but no additional implementation paths are inferred.
Main entry points: streamlit_frontend.py Streamlit interface · searching.py Query search · inverted_index.py Index builder · wiki_corpus.db SQLite corpus
Components 12
Corpus ingestion
- Parse and clean
clean_up_and_connect_to_db.py - SQLite corpus
wiki_corpus.db
Index construction
- Text tokenizer
tokenize_and_stem_pipeline.py - Index builder
inverted_index.py - Inverted index
inverted_index.pkl - Document lengths
doc_lengths.pkl
Search runtime
- Query search
searching.py - BM25 ranking
ranking.py
User interface
- Streamlit interface
streamlit_frontend.py - Displayed results
streamlit_frontend.py
Other components
- Wikipedia dump
- Search user
Connections 17
- Wikipedia dump → Parse and clean: parses
- Parse and clean → SQLite corpus: writes documents
- Index builder → SQLite corpus: reads documents
- Index builder → Text tokenizer: tokenizes text
- Index builder → Inverted index: writes term counts
- Index builder → Document lengths: writes lengths
- Search user → Streamlit interface: enters query
- Streamlit interface → Query search: runs query
- Query search → Text tokenizer: tokenizes query
- Query search → Inverted index: looks up terms
- Query search → BM25 ranking: scores candidates
- BM25 ranking → Inverted index: reads term counts
- BM25 ranking → Document lengths: reads lengths
- Query search → Streamlit interface: returns ranked IDs
- Streamlit interface → SQLite corpus: fetches document details
- Streamlit interface → Displayed results: displays results
- Streamlit interface → SQLite corpus: runs substring query