How nahuelpanigo/document_extraction_llm works
Project type and purpose
• This is a document‐extraction and metadata‐extraction system built in Python. It consists of:
• A data‐preparation pipeline to download, clean, normalize, and structure PDF documents into JSON (download_prepare_clean_normalize_sedici_dataset).
• A fine‐tuning pipeline that generates training tokens, fine‐tunes LLMs on the prepared dataset, and validates the results (fine_tunning + validation).
• A microservice‐based runtime inference API that ingests documents, extracts text, calls an LLM to extract metadata, and orchestrates the results (api/app with extractor_service, llm_service, orchestrator).
File structure
• download_prepare_clean_normalize_sedici_dataset/: data ingestion, CSV filtering, PDF download, OCR/text extraction, JSON generation.
• fine_tunning/ + validation/: token generation, model management, Hugging Face integration, training orchestration, result validation scripts.
• api/app/: three FastAPI microservices, each in its own subfolder (extractor_service, llm_service, orchestrator), plus Dockerfiles and docker-compose.yml.
• utils/ and constants.py: shared utilities for normalization, OCR, terminal coloring.
• Root scripts: run_all_services, run_extractor_temp, etc., for local launches.
Key insights from the README
• The README describes three high‐level flows: GENERATING TRAIN DATA → FINE TUNING → VALIDATION → RUNNING API.
• The API exposes an /upload endpoint for inference.
• It suggests Docker + FastAPI for deployment, plus environment‐based configs.