imamh0ssain/paralegal

1. Paralegal is a document-grounded memo pipeline, not a generic web app. The core flow is: ingest legal/regulatory documents, normalize them into evidence blocks, index those blocks for hybrid retrieval, generate a cited memo from retrieved evidence, then validate citations and evaluate retrieval quality.

2. app.py is the Streamlit presentation layer. It sits on top of the same backend modules as the CLI and is used for document preparation, memo generation, evidence inspection, feedback capture, and evaluation. The UI is a consumer of pipeline artifacts rather than the system of record.

3. paralegal/cli.py is the orchestration boundary. The CLI exposes stage-by-stage commands such as ingest, index, draft, evaluate, download-real, and make-samples, plus the end-to-end all workflow. This makes the repository deployable in both fully local and API-assisted modes.

4. paralegal/ingestion.py is the first major processing subsystem. It handles PDFs, text, markdown, and image files, performs native PDF text extraction before OCR, optionally invokes OCR/layout parsing, and emits normalized evidence records with stable source IDs like DOC2:p6:b1. It also extracts structured fields such as parties, dates, addresses, amounts, signatures, deadlines, and unclear spans.

5. The ingestion output becomes the durable intermediate data layer under data/processed/. The README shows two main artifacts: processed_documents.jsonl for page/block-level evidence and document_summaries.json for document-level summaries. These artifacts are the handoff between extraction and retrieval/generation.

6. paralegal/retrieval.py is the search subsystem. It combines SQLite FTS5 keyword search, vector embeddings, reciprocal-rank fusion, and optional reranking. The resulting index is stored separately under evidence_index/, which is the retrieval boundary used by drafting and evaluation.

7. paralegal/generation.py consumes retrieved evidence plus learned operator feedback patterns to produce a memo draft with citations. The system is explicitly grounded: memo claims should map back to evidence blocks, and unsupported claims are rejected through citation validation rather than being silently emitted.

8. paralegal/feedback.py implements the operator-edit learning loop. It captures before/after edits, stores reusable patterns, and feeds those patterns back into future drafting. This is a distinct subsystem from generation; it models user preference and style changes, not retrieval facts.

9. paralegal/evaluation.py measures the system end-to-end. It reports retrieval metrics such as term recall@5, source recall@5, and MRR, plus extraction counts, warnings, citation validity, unsupported factual lines, and edit-learning persistence. Evaluation outputs are written to sample_outputs/ as both Markdown and JSON reports.

10. paralegal/providers.py and paralegal/local_device.py define the execution backends. Providers can route OCR, embeddings, reranking, and drafting to optional external APIs like OpenAI and Mistral, or to local components such as Docling, Sentence Transformers, BGE reranker, and Ollama. local_device.py controls CPU/CUDA selection so the local stack can run safely on mixed hardware.

11. paralegal/models.py, io_utils.py, text_utils.py, and real_data.py are supporting layers. They define shared data structures, file I/O helpers, text normalization, and public-document acquisition, keeping the major pipeline stages decoupled from parsing and filesystem details.

12. The repository has two operating modes with the same architecture: privacy-preserving local mode and optional API-assisted mode. In local mode, documents, embeddings, reranking, and drafting stay on the machine; in API mode, selected steps can leave the machine when configured. The architecture is therefore a modular RAG pipeline with clear boundaries between ingestion, indexing, retrieval, generation, feedback, and evaluation.

13. tests/ verifies core behavior and local device selection, which is important because the pipeline’s correctness depends on consistent extraction, indexing, and backend routing. The sample artifacts under sample_outputs/ act as reference outputs for the full workflow and are useful when drawing the “inputs → processed evidence → index → memo/report” diagram.