# nisarg-main/github-repo-search-engine- architecture

> Architecture diagram of the GitHub repository nisarg-main/github-repo-search-engine-, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nisarg-main/github-repo-search-engine-
- Repository: https://github.com/nisarg-main/github-repo-search-engine-

## Overview

gitindex is a GitHub repository search engine: it collects repository metadata and READMEs, preprocesses and indexes the corpus, then parses user queries, filters candidates, and ranks results. Users search through either the CLI or Flask UI; a separate evaluation capability compares rankers against relevance judgments. The graph reflects documented capabilities and sampled calls; some crawler and ranker implementation details were omitted from the excerpts, so uncertain internal wiring is not shown.

Main entry points: [`src/query.py`](https://github.com/nisarg-main/github-repo-search-engine-/blob/main/src/query.py) (Query engine), [`run_pipeline.py`](https://github.com/nisarg-main/github-repo-search-engine-/blob/main/run_pipeline.py) (Corpus pipeline), [`src/index.py`](https://github.com/nisarg-main/github-repo-search-engine-/blob/main/src/index.py) (Inverted index), [`src/crawler.py`](https://github.com/nisarg-main/github-repo-search-engine-/blob/main/src/crawler.py) (Repository crawler)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["Corpus collection"]
  node_pipeline["Corpus pipeline<br/>[run_pipeline.py]"]
  node_crawler["Repository crawler<br/>[crawler.py]"]
  node_raw_corpus[("Raw corpus")]
end

subgraph group_indexing["Corpus indexing"]
  node_preprocessor["Text preprocessing<br/>[preprocess.py]"]
  node_processed_corpus[("Processed corpus")]
  node_indexer["Inverted index<br/>[index.py]"]
  node_index_store[("Index store<br/>[inverted_index.pkl]")]
end

subgraph group_retrieval["Search and ranking"]
  node_search_engine["Query engine<br/>[query.py]"]
  node_query_preprocess["Query tokenization<br/>[preprocess.py]"]
  node_tfidf["TF-IDF ranker<br/>[ranking.py]"]
  node_bm25["BM25 ranker<br/>[ranking.py]"]
  node_popularity["Popularity blend<br/>[ranking.py]"]
end

subgraph group_interfaces["User interfaces"]
  node_cli["Search CLI<br/>[search_cli.py]"]
  node_web_app["Flask search app<br/>[app.py]"]
  node_web_page["Search page<br/>[index.html]"]
end

subgraph group_evaluation["IR evaluation"]
  node_evaluator["Ranker evaluation<br/>[evaluate.py]"]
  node_judgments["Relevance judgments"]
end

node_searcher(("Search user"))
node_github["GitHub API"]

node_searcher -->|"submits query"| node_cli
node_searcher -->|"uses search"| node_web_app
node_pipeline -->|"runs crawl"| node_crawler
node_crawler -->|"requests repos"| node_github
node_crawler -->|"fetches READMEs"| node_github
node_crawler -->|"returns corpus"| node_pipeline
node_pipeline -->|"writes corpus"| node_raw_corpus
node_pipeline -->|"reads corpus"| node_raw_corpus
node_pipeline -->|"processes corpus"| node_preprocessor
node_preprocessor -->|"writes documents"| node_processed_corpus
node_pipeline -->|"builds index"| node_indexer
node_indexer -->|"reads documents"| node_processed_corpus
node_indexer -->|"saves index"| node_index_store
node_cli -->|"loads and searches"| node_search_engine
node_web_app -->|"renders page"| node_web_page
node_web_app -->|"runs search"| node_search_engine
node_web_app -->|"loads index"| node_index_store
node_search_engine -->|"loads index"| node_index_store
node_search_engine -->|"tokenizes query"| node_query_preprocess
node_search_engine -->|"dispatches scoring"| node_tfidf
node_search_engine -->|"dispatches scoring"| node_bm25
node_search_engine -->|"dispatches scoring"| node_popularity
node_tfidf -->|"reads postings"| node_indexer
node_bm25 -->|"reads postings"| node_indexer
node_popularity -->|"extends scoring"| node_bm25
node_popularity -->|"reads metadata"| node_indexer
node_evaluator -->|"reads judgments"| node_judgments
node_evaluator -->|"loads index"| node_index_store
node_evaluator -->|"evaluates results"| node_search_engine

click node_pipeline "https://github.com/nisarg-main/github-repo-search-engine-/blob/main/run_pipeline.py"
click node_crawler "https://github.com/nisarg-main/github-repo-search-engine-/blob/main/src/crawler.py"
click node_raw_corpus "https://github.com/nisarg-main/github-repo-search-engine-/blob/main/data/raw/repositories_raw.json"
click node_preprocessor "https://github.com/nisarg-main/github-repo-search-engine-/blob/main/src/preprocess.py"
click node_processed_corpus "https://github.com/nisarg-main/github-repo-search-engine-/blob/main/data/processed/repositories_processed.json"
click node_indexer "https://github.com/nisarg-main/github-repo-search-engine-/blob/main/src/index.py"
click node_index_store "https://github.com/nisarg-main/github-repo-search-engine-/blob/main/data/processed/inverted_index.pkl"
click node_cli "https://github.com/nisarg-main/github-repo-search-engine-/blob/main/search_cli.py"
click node_web_app "https://github.com/nisarg-main/github-repo-search-engine-/blob/main/app.py"
click node_web_page "https://github.com/nisarg-main/github-repo-search-engine-/blob/main/templates/index.html"
click node_search_engine "https://github.com/nisarg-main/github-repo-search-engine-/blob/main/src/query.py"
click node_query_preprocess "https://github.com/nisarg-main/github-repo-search-engine-/blob/main/src/preprocess.py"
click node_tfidf "https://github.com/nisarg-main/github-repo-search-engine-/blob/main/src/ranking.py"
click node_bm25 "https://github.com/nisarg-main/github-repo-search-engine-/blob/main/src/ranking.py"
click node_popularity "https://github.com/nisarg-main/github-repo-search-engine-/blob/main/src/ranking.py"
click node_evaluator "https://github.com/nisarg-main/github-repo-search-engine-/blob/main/src/evaluate.py"
click node_judgments "https://github.com/nisarg-main/github-repo-search-engine-/blob/main/eval/relevance_judgments.json"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_pipeline,node_crawler,node_raw_corpus,node_searcher toneBlue
class node_preprocessor,node_processed_corpus,node_indexer,node_index_store toneAmber
class node_search_engine,node_query_preprocess,node_tfidf,node_bm25,node_popularity,node_github toneMint
class node_cli,node_web_app,node_web_page toneRose
class node_evaluator,node_judgments toneIndigo
```

## Components

### Corpus collection

- [Corpus pipeline](https://github.com/nisarg-main/github-repo-search-engine-/blob/main/run_pipeline.py): `run_pipeline.py`
- [Repository crawler](https://github.com/nisarg-main/github-repo-search-engine-/blob/main/src/crawler.py): `src/crawler.py`
- [Raw corpus](https://github.com/nisarg-main/github-repo-search-engine-/blob/main/data/raw/repositories_raw.json): `data/raw/repositories_raw.json`

### Corpus indexing

- [Text preprocessing](https://github.com/nisarg-main/github-repo-search-engine-/blob/main/src/preprocess.py): `src/preprocess.py`
- [Processed corpus](https://github.com/nisarg-main/github-repo-search-engine-/blob/main/data/processed/repositories_processed.json): `data/processed/repositories_processed.json`
- [Inverted index](https://github.com/nisarg-main/github-repo-search-engine-/blob/main/src/index.py): `src/index.py`
- [Index store](https://github.com/nisarg-main/github-repo-search-engine-/blob/main/data/processed/inverted_index.pkl): `data/processed/inverted_index.pkl`

### Search and ranking

- [Query engine](https://github.com/nisarg-main/github-repo-search-engine-/blob/main/src/query.py): `src/query.py`
- [Query tokenization](https://github.com/nisarg-main/github-repo-search-engine-/blob/main/src/preprocess.py): `src/preprocess.py`
- [TF-IDF ranker](https://github.com/nisarg-main/github-repo-search-engine-/blob/main/src/ranking.py): `src/ranking.py`
- [BM25 ranker](https://github.com/nisarg-main/github-repo-search-engine-/blob/main/src/ranking.py): `src/ranking.py`
- [Popularity blend](https://github.com/nisarg-main/github-repo-search-engine-/blob/main/src/ranking.py): `src/ranking.py`

### User interfaces

- [Search CLI](https://github.com/nisarg-main/github-repo-search-engine-/blob/main/search_cli.py): `search_cli.py`
- [Flask search app](https://github.com/nisarg-main/github-repo-search-engine-/blob/main/app.py): `app.py`
- [Search page](https://github.com/nisarg-main/github-repo-search-engine-/blob/main/templates/index.html): `templates/index.html`

### IR evaluation

- [Ranker evaluation](https://github.com/nisarg-main/github-repo-search-engine-/blob/main/src/evaluate.py): `src/evaluate.py`
- [Relevance judgments](https://github.com/nisarg-main/github-repo-search-engine-/blob/main/eval/relevance_judgments.json): `eval/relevance_judgments.json`

### Other components

- **Search user**
- **GitHub API**

## Connections

- Search user → Search CLI: submits query
- Search user → Flask search app: uses search
- Corpus pipeline → Repository crawler: runs crawl
- Repository crawler → GitHub API: requests repos
- Repository crawler → GitHub API: fetches READMEs
- Repository crawler → Corpus pipeline: returns corpus
- Corpus pipeline → Raw corpus: writes corpus
- Corpus pipeline → Raw corpus: reads corpus
- Corpus pipeline → Text preprocessing: processes corpus
- Text preprocessing → Processed corpus: writes documents
- Corpus pipeline → Inverted index: builds index
- Inverted index → Processed corpus: reads documents
- Inverted index → Index store: saves index
- Search CLI → Query engine: loads and searches
- Flask search app → Search page: renders page
- Flask search app → Query engine: runs search
- Flask search app → Index store: loads index
- Query engine → Index store: loads index
- Query engine → Query tokenization: tokenizes query
- Query engine → TF-IDF ranker: dispatches scoring
- Query engine → BM25 ranker: dispatches scoring
- Query engine → Popularity blend: dispatches scoring
- TF-IDF ranker → Inverted index: reads postings
- BM25 ranker → Inverted index: reads postings
- Popularity blend → BM25 ranker: extends scoring
- Popularity blend → Inverted index: reads metadata
- Ranker evaluation → Relevance judgments: reads judgments
- Ranker evaluation → Index store: loads index
- Ranker evaluation → Query engine: evaluates results