How michniks2001/outline-generator works
1. System shape
- This repo is a two-tier application: a React/Vite frontend and a FastAPI backend.
- The frontend handles document upload, text extraction in the browser, and user interaction.
- The backend owns persistence, OCR fallback, semantic search, and AI-driven outline generation.
2. Frontend boundary
- src/App.jsx is the main orchestration layer for the UI.
- src/Chatbot.jsx and src/SourceModal.jsx suggest the UI is centered on asking questions and inspecting retrieved source chunks.
- src/main.jsx boots the app; styles are split across component-specific CSS files.
3. Client-side document ingestion
- The README says PDFs are first processed in the browser using pdfjs-dist.
- If extracted text is too small, the file is sent to the backend for OCR.
- This creates an important architectural split: lightweight PDFs stay mostly client-side, while scanned/complex PDFs rely on server-side processing.
4. Backend entry and API surface
- backend/app.py is the server entrypoint.
- backend/src/endpoints.py likely defines the HTTP API consumed by the frontend.
- The documented routes show the main backend capabilities:
- /store-text
- /ocr-pdf
- /search-chunks
- /generate-outline
5. Core backend subsystems
- backend/src/database.py indicates a persistence layer, backed by ChromaDB per the README.
- backend/src/models.py likely defines request/response schemas for the API.
- backend/src/utils.py and backend/src/outline.py likely hold reusable text-processing and outline-generation logic.
- backend/src/chatbot.py appears to encapsulate the question-answering / retrieval-augmented generation flow.
6. Document processing pipeline
- Uploaded or OCR-extracted text is chunked before storage.
- The README specifies sentence-aware chunking with ~200-character chunks and ~40-character overlap.
- Large PDFs are processed in batches of 30 pages, which is an operational boundary for throughput and memory usage.
7. Vector search and retrieval
- Chunks are stored in ChromaDB for semantic retrieval.
- The search endpoint returns relevant chunks based on the user’s questions.
- This retrieved context is then reused by the outline generation flow, making search the bridge between stored documents and AI output.
8. AI generation path
- The backend integrates Google Gemini via environment variables in backend/.env.
- backend/src/outline.py likely converts retrieved chunks plus user questions into a structured research-paper outline.
- The model is not just generating freeform chat; it is producing a constrained, outline-shaped response from retrieved evidence.
9. OCR fallback path
- When client-side PDF text extraction is insufficient, the backend uses pdf2image + pytesseract.
- This requires an external Tesseract OCR installation, which is a deployment/runtime dependency.
- Architecturally, OCR is a resilience path for scanned or image-based PDFs, not the default ingestion path.
10. State and storage boundaries
- Persistent document memory lives in the backend’s ChromaDB store, not in the browser.
- The README implies folder-based organization, so document metadata is likely stored alongside chunks to partition content by folder or collection.
- The frontend does not appear to own long-term document state; it is a thin interaction layer over backend storage and retrieval.
11. Operational concerns
- The backend runs on localhost:8000; the frontend on Vite’s dev server, typically localhost:5173.
- CORS must allow the frontend origin to call the API.
- The backend is configured through environment variables for Gemini key and model selection, making the AI provider an external dependency.
12. How to diagram this repo
- A useful architecture diagram would show:
- Browser UI
- Client-side PDF extraction
- OCR fallback to backend
- FastAPI API layer
- Chunking + embedding/storage in ChromaDB
- Retrieval/search
- Gemini-based outline generation
- UI display of outline and source chunks
- The key flow is: PDF/question input → text extraction → chunk storage/search → Gemini outline generation → UI presentation.
Main entry points: endpoints.py HTTP routes · App.jsx Browser UI · outline.py Outline builder · utils.py Chunk text
Components 19
Frontend
React/Vite browser app for upload, question handling, and result display.
- Browser UI · Main interactive client for uploading documents, asking questions, and viewing outputs.
src/App.jsx - App boot · Vite/React entry point that mounts the application.
src/main.jsx - Chat panel · Conversation-style interface for asking questions over stored document content.
src/Chatbot.jsx - Source modal · Popup for inspecting retrieved source chunks and evidence.
src/SourceModal.jsx - PDF extract · Client-side PDF text extraction before falling back to server OCR.
Backend
FastAPI service for storage, OCR fallback, search, and outline generation.
- API server · Backend application entrypoint that starts the FastAPI service.
backend/app.py - HTTP routes · Endpoint layer exposing text storage, OCR, search, and outline generation.
backend/src/endpoints.py - Schemas · Request and response models shared across API handlers.
backend/src/models.py - Store text · Receives extracted text for chunking and persistence.
- OCR fallback · Converts PDF pages to images and runs OCR on scanned or low-text files.
- Chunk text · Sentence-aware splitting with overlap to prepare retrieval units.
backend/src/utils.py - ChromaDB · Persistent semantic store for document chunks and metadata.
backend/src/database.py - Search chunks · Semantic lookup over stored chunks for a user question.
- QA engine · Retrieval-augmented answering flow that assembles context for responses.
backend/src/chatbot.py - Outline builder · Produces a structured outline from retrieved evidence and model prompts.
backend/src/outline.py - Generate outline · API route that returns an AI-generated outline for the selected topic or document.
- Gemini API · External model used for outline generation and AI-assisted responses.
- Env config · Backend environment variables for API keys, model selection, and service settings.
backend/.env.example - Tesseract OCR · External OCR engine required for server-side document recognition.
Connections 23
- App boot → Browser UI: mounts · Bootstraps the main application.
- Browser UI → PDF extract: extracts · Processes PDFs locally when text extraction is sufficient.
- Browser UI → Chat panel: renders · Primary interaction surface for questions and answers.
- Browser UI → Source modal: opens · Shows source evidence for retrieved chunks.
- Browser UI → HTTP routes: calls · Sends upload, search, and outline requests to the backend.
- API server → HTTP routes: serves · Starts the HTTP API layer.
- HTTP routes → Schemas: uses · Validates incoming and outgoing payloads.
- HTTP routes → Store text: exposes · Endpoint for persisting extracted text.
- HTTP routes → OCR fallback: exposes · Endpoint for OCR fallback on scanned PDFs.
- HTTP routes → Search chunks: exposes · Endpoint for semantic retrieval over stored chunks.
- HTTP routes → Generate outline: exposes · Endpoint for structured outline generation.
- Store text → Chunk text: routes to · Breaks uploaded text into retrieval-sized units.
- OCR fallback → Tesseract OCR: depends on · Uses OCR engine for image-based text extraction.
- OCR fallback → Chunk text: feeds · OCR output is normalized and chunked before storage.
- Chunk text → ChromaDB: writes · Persists chunks for later semantic search.
- Search chunks → ChromaDB: queries · Retrieves relevant chunks by similarity.
- Search chunks → QA engine: supplies context · Provides evidence chunks for question answering.
- QA engine → Gemini API: invokes · Uses the model to synthesize answers from retrieved context.
- Generate outline → Outline builder: delegates · Hands off outline construction to the generation pipeline.
- Outline builder → Search chunks: retrieves evidence · Pulls supporting chunks before generation.
- Outline builder → Gemini API: invokes · Uses the model to produce structured outline output.
- Env config → Gemini API: configures · Provides API keys and model selection.
- Env config → API server: configures · Supplies backend runtime settings.