How tirthkothari/preciseextract works
Project type and purpose
• This is a full-stack web application (“Precise Extract”) for bulk extraction of tabular data from PDFs/images, storage in MySQL, natural-language querying via Gemini/ChatGPT, and Excel export.
• It uses a Flask “main” server for user interaction (signup/login, upload, query UI) and a FastAPI microservice for file ingestion, OCR/extraction, and storage.
File structure
• Top-level folders:
• Backend/Precise_extract – Flask app (routes.py, forms.py, templates/, static/)
• fastapi – FastAPI service (myapp.py, UPLOADED_FILES/, Extract/PaddleOCR/)
• DATASET – sample JSON/images
• precise_extract.sql – database schema
• requirements.txt, LICENSE, README.md
• Flask server entrypoint: run.py
• FastAPI entrypoint: myapp.py (serving on port 8000, JWT/AES authentication)
• Templates & static assets define the frontend UI pages (login, upload, browse, query, export).
• “UPLOADED_FILES” holds user uploads; “Extract/PaddleOCR” handles OCR logic.
README insights
• Workflow steps: User → File upload → FastAPI (OCR via PaddleOCR) → MySQL storage → Query (Flask UI calls Gemini/ChatGPT → SQL) → Export.
• Two backend services: Flask (port 5000) for UI and auth, FastAPI (port 8000) for ingestion and extraction.
• Uses MySQL, PaddleOCR, Gemini/ChatGPT API.
Main components & relationships
a. Frontend (Flask templates + static JS/CSS)
b. Flask backend
– Routes for signup/login, upload UI, query UI, Excel export
– Calls FastAPI endpoints for file ingestion (via HTTP + JWT token)
– Calls external API (Gemini/ChatGPT) for NL→SQL conversion
c. FastAPI microservice
– Receives file uploads, applies AES decryption/JWT auth
– Converts PDF→images, invokes PaddleOCR module to extract tables
– Writes extracted rows to MySQL
d. MySQL database
e. External services
– PaddleOCR (local library)
– Gemini/ChatGPT API for natural-language query translation
f. Storage
– UPLOADED_FILES folder (temporary file storage)
– DATASET folder (sample data for testing)
Data flows:
1) User → Flask UI → HTTP POST to FastAPI /upload
2) FastAPI → OCR → MySQL insert
3) User submits query → Flask → Gemini/ChatGPT → SQL → MySQL → results
4) User requests export → Flask → reads MySQL → generate Excel → frontend download
Architectural patterns & design principles
• Separation of concerns: UI/auth server (Flask) vs ingestion service (FastAPI)
• Microservice-style: FastAPI service dedicated to heavy OCR/extraction tasks
• Stateless HTTP communication with JWT/AES for securing file uploads
• Modular extraction logic under fastapi/Extract/PaddleOCR
• MVC-like structure in Flask (templates/views/forms/models)
• Data store details (MySQL schema: precise_extract.sql)
• Validate with the team that flows match actual implementation (e.g., template names, API endpoints).