How nisarg-main/github-repo-search-engine- works
gitindex is a GitHub repository search engine: it collects repository metadata and READMEs, preprocesses and indexes the corpus, then parses user queries, filters candidates, and ranks results. Users search through either the CLI or Flask UI; a separate evaluation capability compares rankers against relevance judgments. The graph reflects documented capabilities and sampled calls; some crawler and ranker implementation details were omitted from the excerpts, so uncertain internal wiring is not shown.
Main entry points: query.py Query engine · run_pipeline.py Corpus pipeline · index.py Inverted index · crawler.py Repository crawler
Components 19
Corpus collection
- Corpus pipeline
run_pipeline.py - Repository crawler
src/crawler.py - Raw corpus
data/raw/repositories_raw.json
Corpus indexing
- Text preprocessing
src/preprocess.py - Processed corpus
data/processed/repositories_processed.json - Inverted index
src/index.py - Index store
data/processed/inverted_index.pkl
Search and ranking
- Query engine
src/query.py - Query tokenization
src/preprocess.py - TF-IDF ranker
src/ranking.py - BM25 ranker
src/ranking.py - Popularity blend
src/ranking.py
User interfaces
- Search CLI
search_cli.py - Flask search app
app.py - Search page
templates/index.html
IR evaluation
- Ranker evaluation
src/evaluate.py - Relevance judgments
eval/relevance_judgments.json
Other components
- Search user
- GitHub API
Connections 29
- Search user → Search CLI: submits query
- Search user → Flask search app: uses search
- Corpus pipeline → Repository crawler: runs crawl
- Repository crawler → GitHub API: requests repos
- Repository crawler → GitHub API: fetches READMEs
- Repository crawler → Corpus pipeline: returns corpus
- Corpus pipeline → Raw corpus: writes corpus
- Corpus pipeline → Raw corpus: reads corpus
- Corpus pipeline → Text preprocessing: processes corpus
- Text preprocessing → Processed corpus: writes documents
- Corpus pipeline → Inverted index: builds index
- Inverted index → Processed corpus: reads documents
- Inverted index → Index store: saves index
- Search CLI → Query engine: loads and searches
- Flask search app → Search page: renders page
- Flask search app → Query engine: runs search
- Flask search app → Index store: loads index
- Query engine → Index store: loads index
- Query engine → Query tokenization: tokenizes query
- Query engine → TF-IDF ranker: dispatches scoring
- Query engine → BM25 ranker: dispatches scoring
- Query engine → Popularity blend: dispatches scoring
- TF-IDF ranker → Inverted index: reads postings
- BM25 ranker → Inverted index: reads postings
- Popularity blend → BM25 ranker: extends scoring
- Popularity blend → Inverted index: reads metadata
- Ranker evaluation → Relevance judgments: reads judgments
- Ranker evaluation → Index store: loads index
- Ranker evaluation → Query engine: evaluates results