How jzhang512/metascribe works
Project Type and Purpose
• MetaScribe is a command-line tool designed to automate the extraction and generation of structured descriptive metadata from digital collections. It leverages large language models (LLM) and supplementary techniques such as OCR to process documents page-by-page.
• Its purpose is to allow users (typically libraries, archives, or digital repositories) to define a custom metadata schema (via JSON Schema) and run a configurable extraction pipeline that produces JSONL-formatted metadata output.
File Structure
• Top-Level Files:
• metascribe_config.yaml: Likely a global configuration file indicating extraction settings.
• run.py: Acts as the entry point to the application, parsing command-line arguments (input directories, custom configurations, and output location).
• requirements.txt: Lists Python dependencies.
• Directories:
• ppa_examples: Contains sample JSON configurations that serve as examples or templates for the custom metadata schemas.
• src: This is where the core source code resides.
• controller.py: Likely responsible for orchestration or coordinating between input, processing, and output steps.
• core (subdirectory): Contains the functional components:
▪ aggregation.py: Handles aggregation of metadata fields defined in the JSON Schema.
▪ generate_metadata.py: Likely builds the metadata output from processed inputs.
▪ llm_interface.py: Provides an abstraction for interacting with the supported large language models.
▪ ocr.py: Provides functionality to perform OCR, extracting text from document images.
▪ preprocessing_document.py: Prepares or pre-processes documents before passing to OCR or subsequent stages.
• src/resources:
• ZigZag.jar: This JAR file suggests an integration with a Java-based component or tool possibly used for additional processing/analysis.
Architectural Patterns and Pipeline
• The project adopts a modular, pipeline-based design where each component focuses on a specific phase in the metadata extraction process:
• Input/Configuration: run.py reads the input directory along with any custom configuration (metascribe_config.yaml or examples in ppa_examples).
• Preprocessing: preprocessing_document.py prepares raw input data.
• OCR Processing: ocr.py extracts text from document pages.
• LLM Interaction: llm_interface.py sends data to external LLMs (OpenAI, Anthropic, etc.) for metadata extraction.
• Metadata Generation/Aggregation: generate_metadata.py and aggregation.py use the responses to build structured metadata.
• Controller (controller.py) ties these steps together, managing control flow and error handling (as documented, errors are logged in a manifest.json, and processing can resume).
• A configuration component (using the YAML file and JSON examples) that tailors the extraction schema.
• Follow with the pipeline stages:
• Preprocessing (document formatting and cleaning)
• OCR Module (text extraction)
• LLM Interface (communication with external language models, showing external API dependencies such as OpenAI, Anthropic, etc.)
• Metadata Generator and Aggregator (including the core logic that produces JSONL output).
• External integration of ZigZag.jar from the resources folder (possibly as an auxiliary processing module).
• Specify the output module that writes metadata to files and maintains the run status in manifest.json.
• If applicable, note error-handling paths and the “resume processing” feedback loop.