How scriptedbyarun47/kmrl_docsummarizer works
1. Repository purpose
- This is a document summarization system for policy and HR/resume PDFs. The repo is organized as a pipeline that ingests source documents, extracts text, generates summaries, and then serves or presents those summaries through a separate backend/frontend stack.
2. Source document layer
- app/data/raw/ holds the original PDFs.
- app/data/processed/ holds extracted text versions of those PDFs.
- app/data/summaries/ holds the generated summary text files.
- This makes the filesystem itself an important artifact store in the architecture, not just a temporary workspace.
3. Python processing pipeline
- The core document workflow lives under app/scripts/.
- ingestion.py likely handles loading files into the pipeline.
- extraction.py is the text extraction stage from PDF to plain text.
- document_summarizer.py is the summarization stage.
- utils.py and config.py provide shared helpers and environment/configuration.
4. Python entrypoint and orchestration
- app/main.py is the likely orchestration entrypoint for the Python side.
- The app package structure suggests the document workflow can run as a local batch process or CLI-style job, rather than only as a service.
5. Persistence and model boundary
- app/scripts/db.py indicates the Python pipeline can interact with some form of storage layer beyond the filesystem.
- On the Node side, backend/models/Summary.js suggests a persisted summary record model, likely used to store or retrieve summary metadata/content through the backend.
- Together, these point to two storage surfaces: file-based artifacts for processing, and a backend data model for application access.
6. Node.js backend service
- backend/server.js is the runtime server layer.
- Its role is likely to expose the summarized document data to clients, separating presentation/API concerns from the Python processing pipeline.
- The backend appears to be the bridge between stored summaries and the frontend.
7. Frontend presentation layer
- frontend/index.html, frontend/app.js, and frontend/styles.css form a lightweight browser UI.
- This layer is responsible for displaying documents or summaries and likely calls the Node backend for data.
- It is a distinct consumer of the backend rather than part of the document-processing pipeline.
8. Operational artifacts
- logs/process.log captures pipeline or application execution logs.
- This indicates the summarization workflow is operationalized enough to track processing outcomes, errors, or document runs.
9. Experimentation and validation
- notebooks/test_summary_generation.ipynb shows that summary quality was explored interactively.
- This notebook sits outside the production path and is likely used to tune or validate extraction/summarization behavior.
10. Technology split and dependency boundaries
- The repo uses Python for document processing (requirements.txt) and Node.js for the application server (package.json, package-lock.json).
- That split is a key architectural boundary: offline or batch-style text processing in Python, application serving and UI delivery in JavaScript.
11. End-to-end data flow
- The most likely flow is: PDF enters app/data/raw/ → Python extracts text into app/data/processed/ → Python generates summary into app/data/summaries/ → backend serves summary data via the model/server layer → frontend displays it to the user.
- That flow is the central diagram to draw for this repository.
Main entry points: document_summarizer.py Summarizer · ingestion.py Ingestion · processed Extracted text · utils.py Utilities
Components 17
Python pipeline
Document ingestion, extraction, summarization, and local orchestration.
- Ingestion · Loads source documents into the processing flow.
app/scripts/ingestion.py - Extraction · Converts PDFs into text.
app/scripts/extraction.py - Summarizer · Generates summaries from extracted text.
app/scripts/document_summarizer.py - Main runner · Batch orchestration entrypoint for the pipeline.
app/main.py - Config · Shared configuration and environment settings.
app/scripts/config.py - Utilities · Shared helpers used across pipeline stages.
app/scripts/utils.py - DB access · Python-side persistence or storage integration.
app/scripts/db.py
Artifact store
Filesystem-based inputs, intermediates, summaries, and logs.
- Raw PDFs · Source policy and resume documents.
app/data/raw - Extracted text · Plain-text outputs from PDF extraction.
app/data/processed - Summaries · Generated summary text files.
app/data/summaries
Backend service
Node runtime and persisted summary model.
- Summary model · Persisted summary record used by the backend.
backend/models/Summary.js - Server · Application server exposing summary data.
backend/server.js
Frontend UI
Browser presentation layer for summaries.
- Browser UI · Client-side display for documents and summaries.
frontend/app.js - Shell · Static page container for the UI.
frontend/index.html - Styles · Presentation styling for the browser UI.
frontend/styles.css
Ops and experiments
Logging and notebook-based validation.
- Process log · Execution log for pipeline or app runs.
logs/process.log - Summary notebook · Interactive notebook for testing summary generation.
notebooks/test_summary_generation.ipynb
Connections 21
- Main runner → Ingestion: starts · Main runner kicks off document ingestion.
- Ingestion → Raw PDFs: reads · Ingestion consumes source PDFs.
- Ingestion → Extraction: hands off · Loaded documents move into extraction.
- Extraction → Extracted text: writes · Extraction stores plain-text artifacts.
- Extracted text → Summarizer: feeds · Extracted text becomes summarizer input.
- Summarizer → Summaries: writes · Summaries are persisted as text artifacts.
- Config → Main runner: configures · Shared settings shape pipeline execution.
- Utilities → Ingestion: supports · Common helpers are reused during ingest.
- Utilities → Extraction: supports · Common helpers are reused during extraction.
- Utilities → Summarizer: supports · Common helpers are reused during summarization.
- Summarizer → DB access: persists · Summaries can be stored through the data layer.
- DB access → Summary model: maps to · Python storage boundary aligns with backend summary records.
- Summary model → Server: served by · Backend server exposes persisted summary data.
- Server → Browser UI: api/data · Frontend consumes summary data from the backend.
- Browser UI → Shell: renders in · Browser app is mounted in the HTML shell.
- Browser UI → Styles: styled by · Stylesheet controls presentation.
- Main runner → Process log: logs · Pipeline execution is recorded in process logs.
- Summary notebook → Summarizer: validates · Notebook experiments inform summary generation.
- Summary notebook → Extracted text: inspects · Notebook likely inspects extracted text artifacts.
- Raw PDFs → Extracted text: transforms to · PDFs are converted into extracted text.
- Summaries → Server: exposed via · Generated summaries are surfaced through the backend.