How landing-ai/agentic-doc works
Project type and purpose
• This is a Python client library (agentic-doc) that wraps the LandingAI Agentic Document Extraction REST API.
• Its goal is to let downstream Python code call a single parse(...) function (or specialized helpers) to extract structured data from PDFs/images/URLs, handle large‐file splitting, parallelism, retries, and offer utilities (visualizers, connectors).
File structure
• Top-level files:
• pyproject.toml: build and dependency config
• .github/workflows/ci_cd.yml: CI/CD pipeline
• README.md, LICENSE, .flake8, .gitignore: docs and linting
• agentic_doc/ (the library source)
• common.py – core data models, error schemas
• config.py – environment/.env driven settings (batch size, retry caps, logging style)
• connectors.py – abstraction over document sources (Local, URL, S3, Google Drive)
• parse.py – main API surface: parse, parse_documents, parse_and_save…; splits large docs, thread pool orchestration, stitches results
• tests/ – unit & integration tests validating connectors, parse logic, utils, config, common.
README for additional insights
• Emphasis on:
• Single-call API to parse local files, URLs, raw bytes
• Automatic PDF splitting + parallel processing + retry with backoff
• Connectors to abstract away different storage backends
• Utilities for saving groundings and visual debugging
a. Main components to represent:
1. Client Application (your Python code that does “from agentic_doc.parse import parse”)
2. agentic_doc Library (five sub-modules)
– API Layer (parse, parse_documents)
– Connector Layer (LocalConnector, S3Connector, GDriveConnector, URLConnector)
– Processing Layer (PDF splitter, thread‐pool executor, retry mechanism)
– Common/Data Model Layer (Pydantic schemas, chunk & grounding models)
– Utility Layer (viz_parsed_document, save groundings)
3. External Dependencies
– LandingAI Agentic Document Extraction REST API
– OpenCV (cv2) for image reading
– AWS/GCP if using S3/GDrive connectors
4. Configuration / Env Var Store (.env, os.environ)
b. Relationships & interactions (data flow):
1. Client → calls parse()
2. parse() reads Settings from config.py (env or .env)
3. parse() uses Connector(s) to fetch raw bytes / file paths / URLs
4. Processing Layer splits large PDFs (if needed)
5. Thread‐pool: workers perform HTTP calls with retry/backoff → AgenticDoc API
6. Responses aggregated, stitched into hierarchical JSON (common.py models)
7. Results returned to Client
8. Optional: utils.viz_parsed_document → produce annotated images
c. Architectural patterns & principles:
• Layered architecture: API, Connectors, Processing, Utilities
• Strategy pattern for connectors
• Builder/Data Mapper for Pydantic models (common.py)
• Concurrency via thread pool + batching
• Resilience: retry with exponential backoff, jitter
d. Key technologies & libraries:
• Python 3.9+
• requests or httpx (for REST calls)
• concurrent.futures.ThreadPoolExecutor
• Pydantic (for result schemas)
• OpenCV-Python (cv2)
• AWS SDK (boto3) / Google API Client
• dotenv (for .env support)