How adityasohner/repoir works
1. Repository shape
- This is a full-stack monorepo with a Python backend at the root and a separate React frontend in RepoIR Frontend/.
- The backend is the core product: it handles ingestion, indexing, search, auth, storage, and AI orchestration.
- The frontend is a TypeScript/Vite SPA that presents the search dashboard, upload flow, and account/settings screens.
2. Backend entrypoint and service boundary
- The backend appears to start from app_server.py with main.py as an additional bootstrap/entry module.
- The backend is organized as a single API service with internal subsystems rather than multiple microservices.
- The frontend talks to this service through src/lib/api.ts, so the main runtime boundary is browser SPA ↔ FastAPI backend.
3. Ingestion pipeline
- The ingestion flow is split into distinct stages under app/ai/pipeline/:
- extraction from source data,
- normalization,
- chunking,
- embedding generation,
- indexing/storage.
- This pipeline is designed for multiple source types: documents, images, URLs, and raw text snippets.
4. Source-specific extraction layer
- app/ai/extractors/ contains specialized extractors for documents, images, URLs, OCR, and vision description.
- Images are not just stored; they are turned into searchable semantic text via OCR and/or vision captioning.
- This is what enables concept search over photos and scanned content, not just literal filename matching.
5. Text preparation and semantic indexing
- app/ai/normalizers/ cleans and formats content by file type before chunking.
- app/ai/chunker/text_chunker.py splits content into overlapping semantic chunks.
- app/ai/embeddings/text_embedder.py converts chunks into vector embeddings, which are then used for semantic retrieval.
6. Search architecture
- Search is two-stage, as described in the README and reflected by app/ai/pipeline/search_pipeline.py.
- Stage 1 combines keyword retrieval and vector retrieval:
- SQLite FTS5 for lexical/BM25-style lookup,
- FAISS for vector similarity search,
- fusion via Reciprocal Rank Fusion.
- Stage 2 applies an LLM-based relevance filter/re-ranker using SambaNova-hosted Llama 3.1 8B.
7. Storage subsystem
- app/storage/ is the persistence boundary and includes:
- db_store.py for metadata/index records,
- faiss_store.py for vector index persistence,
- object_store.py for file/object handling,
- gdrive_store.py for the Google Drive-backed vault,
- encryption_manager.py for protecting stored content,
- id_generator.py for internal identity management.
- The architecture separates original file storage, metadata/index storage, and vector storage.
8. Cloud vault and external integrations
- A key external dependency is Google Drive, used as the “RepoIR_Vault” for original files.
- OAuth-based access is handled in app/auth/, and the README indicates user password/JWT-based session management as well.
- Another external service is SambaNova API, which powers LLM reranking rather than local model hosting.
9. Auth and security boundaries
- app/auth/manager.py and app/auth/validator.py define the backend authentication layer.
- app/config.py centralizes environment-driven configuration, including API keys and secrets.
- The design emphasizes privacy by keeping chunking, metadata indexing, and retrieval logic inside the private deployment boundary, with only necessary integrations leaving the system.
10. Frontend structure
- The frontend is a Vite/React/TypeScript app with Tailwind CSS and a shadcn-style component library under src/components/ui/.
- src/pages/ contains the main app surfaces: landing, login/signup, dashboard, profile, settings, password recovery, and OAuth callback.
- src/components/ contains dashboard-specific building blocks such as file grid/cards, preview modal, upload zone, sidebar, top nav, stats cards, search/command palette, and layout wrappers.
11. Frontend state and backend interaction
- src/contexts/AuthContext.tsx and src/contexts/AppContext.tsx hold shared auth/app state across the SPA.
- src/lib/api.ts is the main client-side integration point to the backend.
- This suggests the UI is mostly a thin presentation layer over backend-managed data, search, and session state.
12. Dashboard and analytics flow
- The dashboard is not just a file browser; it includes analytics and activity visibility.
- Components like StatsCards, FileGrid, ImmersiveSearch, and FilePreviewModal imply a user flow of upload → indexing → search → preview → manage.
- The README’s “real-time analytics” maps to dashboard views for storage usage, category breakdowns, and recent activity.
13. Deployment and operational tooling
- The repo is containerized with a root Dockerfile, and .github/workflows/ci.yml plus security.yml indicate automated CI and security checks.
- The frontend has a public/_redirects file, which usually supports SPA routing on static hosts.
- Frontend testing uses Vitest; the backend uses Python dependency management via requirements.txt.
14. How to draw the architecture
- A good diagram would place the React SPA on one side, the FastAPI backend in the center, and Google Drive / SambaNova as external services on the edge.
- Inside the backend, show the flow from ingestion/extraction into normalization/chunking/embeddings, then into SQLite metadata + FAISS vectors + encrypted vault storage.
- Search should be shown as a separate path: query → hybrid retrieval → rerank → ranked results back to UI.
Main entry points: ingestion_pipeline.py Ingestion pipeline · app_server.py API server · manager.py Auth manager · search_pipeline.py Search pipeline
Components 25
Frontend SPA
React/Vite client for search, uploads, and account management
- SPA entry · Bootstraps the browser application and routing
RepoIR Frontend/src/main.tsx - Pages · Landing, auth, dashboard, profile, settings, and callback screens
RepoIR Frontend/src/pages/DashboardPage.tsx - Dashboard UI · Layout, search, upload, preview, stats, and navigation building blocks
RepoIR Frontend/src/components/DashboardLayout.tsx - App state · Shared application state across dashboard flows
RepoIR Frontend/src/contexts/AppContext.tsx - Auth state · Session and identity state for the SPA
RepoIR Frontend/src/contexts/AuthContext.tsx - API client · Main client-side integration point to the backend service
RepoIR Frontend/src/lib/api.ts
Backend Service
Python API service with auth, ingestion, search, and storage subsystems
- API server · Primary backend process boundary and request entry
app_server.py - Bootstrap · Additional backend launch and wiring entry
main.py - Runtime config · Central environment-driven configuration for secrets and integrations
app/config.py - Auth manager · Session, password, and OAuth-oriented authentication logic
app/auth/manager.py - Auth validator · Token and credential validation boundary
app/auth/validator.py - Ingestion pipeline · End-to-end pipeline from source input to indexed content
app/ai/pipeline/ingestion_pipeline.py - Source extractors · Document, image, URL, OCR, and vision extraction for searchable text
app/ai/extractors/base.py - Normalize content · File-type-specific cleanup and formatting before chunking
app/ai/normalizers/common.py - Chunk text · Overlapping semantic chunking of normalized content
app/ai/chunker/text_chunker.py - Create vectors · Turns chunks into semantic embeddings for retrieval
app/ai/embeddings/text_embedder.py - Search pipeline · Two-stage search with lexical plus vector retrieval and reranking
app/ai/pipeline/search_pipeline.py
Storage Layer
Persistence and vault boundaries for metadata, vectors, files, and encryption
- Metadata DB · Metadata and index records, including lexical search support
app/storage/db_store.py - Vector index · Persistent vector similarity index
app/storage/faiss_store.py - Object store · File and object handling boundary for persisted content
app/storage/object_store.py - Vault store · Google Drive-backed vault for original files
app/storage/gdrive_store.py - Encryption · Protects stored content at rest
app/storage/encryption_manager.py - ID generator · Internal identity generation for stored records
app/storage/id_generator.py
External Services
Third-party services used for vault storage and LLM reranking
- Google Drive · External vault used for original file storage
- SambaNova LLM · Hosted Llama 3.1 model used for relevance filtering and reranking
Connections 29
- SPA entry → Pages: routes · Application entry renders route-level screens
- Pages → Dashboard UI: composes · Pages are built from dashboard and account components
- Dashboard UI → App state: reads state · UI components consume shared application state
- Dashboard UI → Auth state: reads auth · UI components depend on session state
- App state → API client: calls · App-level actions are fulfilled through backend APIs
- Auth state → API client: auth calls · Login, signup, recovery, and session flows use the API client
- API server → Bootstrap: starts · Primary server process is initialized by bootstrap wiring
- API server → Auth manager: uses · Backend request handling depends on authentication services
- API server → Ingestion pipeline: ingests · Upload and indexing requests enter the ingestion pipeline
- API server → Search pipeline: searches · Query requests are handled by the search pipeline
- API server → Runtime config: loads · Runtime settings and secrets are injected from config
- Auth manager → Auth validator: validates · Authentication manager delegates trust checks to validators
- Auth manager → Runtime config: reads secrets · Auth depends on configured credentials and tokens
- Ingestion pipeline → Source extractors: extracts · Source-specific extraction produces canonical text
- Ingestion pipeline → Normalize content: normalizes · Extracted content is cleaned by file type
- Ingestion pipeline → Chunk text: chunks · Normalized content is split into overlapping semantic chunks
- Ingestion pipeline → Create vectors: embeds · Chunks are converted into vector representations
- Ingestion pipeline → Metadata DB: records · Ingestion writes metadata and index records
- Ingestion pipeline → Vector index: indexes · Embeddings are persisted in the vector index
- Ingestion pipeline → Object store: stores · Processed file objects are retained in the object layer
- Ingestion pipeline → Vault store: uploads · Original files are sent to the Google Drive vault
- Ingestion pipeline → Encryption: encrypts · Stored content is protected before persistence
- Ingestion pipeline → ID generator: assigns ids · Stable identifiers are generated for stored assets
- Search pipeline → Metadata DB: FTS5 lookup · Lexical retrieval uses SQLite full-text search
- Search pipeline → Vector index: vector lookup · Semantic retrieval uses FAISS similarity search
- Search pipeline → SambaNova LLM: reranks · Candidate results are filtered and reranked by the hosted LLM
- Vault store → Google Drive: backed by · Vault storage is implemented on top of Google Drive
- API client → API server: requests · Browser SPA communicates with the backend service
- Auth manager → Auth state: session data · Authentication responses feed SPA session state