How adityak74/pdf-extractor-api works
Project Type and Purpose
• This is a full-stack API application focused on extracting content from PDF files.
• The FastAPI framework is used to expose RESTful endpoints, and there’s heavy use of PDF processing libraries (PyMuPDF, pdfplumber, Pillow) along with SQLAlchemy and PostgreSQL for persistence.
• Background workers (APScheduler, file_cleanup in workers) add support for automatic cleanup tasks.
File Structure
• The top-level structure includes configuration files (Dockerfile, docker-compose.yml, .env.example), migration scripts (alembic), tests, and directories for uploads.
• The core application is located in the “app” folder, which is split into several components:
• Controllers (app/controllers): Responsible for handling API endpoint requests (pdf_controller, worker_controller).
• Services (app/services): Houses business logic (pdf_service).
• Database (app/database): Contains connection setup, ORM models, and repository patterns for persistence.
• Models (app/models): Defines request and response schemas.
• Workers (app/workers): Implements background processing (file_cleanup).
• Utils (app/utils): Contains helper functions (file_utils, check_config).
• The “main.py” in the “app” directory is the application’s entry point.
• Migration scripts are maintained under “alembic”, which shows database versioning is key.
• The “uploads” folder acts as a file storage location for both PDFs and image outputs.
a. Main Components:
• API Layer: Represent the FastAPI “app” with endpoints defined via the controllers.
• Business Logic Layer: The services, specifically pdf_service that performs extraction logic.
• Database Layer: Incorporate the SQLAlchemy models, connection, and repository from “app/database”, plus align to the Alembic migration mechanism.
• Background Workers: Include the file_cleanup worker process from “app/workers”.
• Utility Components: Represent file and configuration utilities.
• External Dependencies: PDF parsing libraries (PyMuPDF, pdfplumber, Pillow) along with the PostgreSQL database.
• Deployment and Infrastructure: Docker, docker-compose configuration, and environment management for containerized deployments.
b. Interactions and Data Flow:
• Controllers delegate to services where PDF extraction processing occurs.
• The service layer then interacts with the database layer for storing/retrieving metadata and extracted content.
• Background workers should be shown as periodically triggered (via APScheduler or similar mechanism) to perform tasks like cleaning up old files.
• Utility modules support multiple layers with helper functions.
• The external storage for uploads (PDFs and images) should be depicted, alongside the direct dependencies on third-party libraries.
c. Architectural Patterns and Design Principles:
• A layered, modular structure: separation of concerns between routing (controllers), business logic (services), and data persistence (database/repository).
• Incorporate a REST API pattern.
• That dependency injection might be in use via FastAPI.
• Background job management is part of the reliability design.
d. Technologies, Frameworks, and Libraries:
• FastAPI for the core API.
• Uvicorn as the ASGI server.
• SQLAlchemy (with Alembic) for ORM and migrations, connected to PostgreSQL.
• PDF extraction libraries: PyMuPDF, pdfplumber, Pillow.
• APScheduler for scheduling background cleanup tasks.
• Docker/Docker Compose for containerization.