# jimmc414/document_intelligence architecture

> Architecture diagram of the GitHub repository jimmc414/document_intelligence, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jimmc414/document_intelligence
- Repository: https://github.com/jimmc414/document_intelligence

## Overview

##### Project Type and Purpose
This is a document processing and analysis system that handles multiple types of unstructured data (PDFs, emails, audio) and processes them through various AI/ML pipelines to extract, analyze, and categorize information.

##### Main System Components

A. Input Layer
- Document Intake (PDFs, emails, audio files)
- File Management System

B. Processing Pipeline Layer
- Text Extraction Services
- PDF Processing
- OCR Processing
- Audio Transcription
- Email Processing

C. Analysis Layer
- Text Analysis Services
- Named Entity Recognition
- Key-Value Extraction
- Sentiment Analysis
- Text Summarization
- Topic Modeling

D. Classification Layer
- Document Classification
- Clustering Services
- Similarity Analysis

E. Integration Layer
- External System Integration
- Account Assignment System

**[Input Sources] → [Processing Pipeline] → [Analysis Engine] → [Classification System] → [Integration Layer]**

##### Input Layer
**[PDF Files] [Email Attachments] [Audio Files]**
- ↓           ↓               ↓
- [File Management System]

##### Processing Pipeline
**[Text Extraction Service]**
- PDF Processor
- OCR Engine
- Audio Transcriber
- Email Processor

##### Analysis Engine
**[Text Analysis Pipeline]**
- NER Service
- KV Extraction
- Sentiment Analyzer
- Summarization Service
- Topic Modeler

##### Classification System
**[Document Classification]**
- Clustering Service
- Similarity Analysis
- Category Assignment

##### Integration Layer
**[External System Integration]**
- Account Assignment Service

##### Technical Details
- Configuration points
- Where model training occurs
- API integration points
- Data persistence points

Remember to indicate the scalability aspects and how the system handles different document types through various processing pipelines while maintaining a unified output format for the external system integration.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Input
        PDF[PDF Files]:::input
        Email[Email Attachments]:::input
        Audio[Audio Files]:::input
        FMS[File Management System]:::input
        click FMS "https://github.com/jimmc414/document_intelligence/blob/main/manage_files.py"
    end

    subgraph Processing
        TE[Text Extraction Services]:::processing
        
        PDF_P[PDF Processing]:::processing
        click PDF_P "https://github.com/jimmc414/document_intelligence/blob/main/extract_text_from_pdf.py"
        
        OCR[OCR Processing]:::processing
        click OCR "https://github.com/jimmc414/document_intelligence/blob/main/optical_character_recognition.py"
        
        Audio_P[Audio Processing]:::processing
        click Audio_P "https://github.com/jimmc414/document_intelligence/blob/main/extract_text_from_audio.py"
        
        Email_P[Email Processing]:::processing
        click Email_P "https://github.com/jimmc414/document_intelligence/blob/main/dl_email.py"
    end

    subgraph Analysis
        NER[Named Entity Recognition]:::analysis
        click NER "https://github.com/jimmc414/document_intelligence/blob/main/extract_named_entities.py"
        
        KV[Key-Value Extraction]:::analysis
        click KV "https://github.com/jimmc414/document_intelligence/blob/main/extract_key_value_pairs.py"
        
        SA[Sentiment Analysis]:::analysis
        click SA "https://github.com/jimmc414/document_intelligence/blob/main/sentiment_analysis.py"
        
        TS[Text Summarization]:::analysis
        click TS "https://github.com/jimmc414/document_intelligence/blob/main/summarize_text.py"
        
        TM[Topic Modeling]:::analysis
        click TM "https://github.com/jimmc414/document_intelligence/blob/main/create_topic_model.py"
    end

    subgraph Classification
        DC[Document Classification]:::classification
        click DC "https://github.com/jimmc414/document_intelligence/blob/main/document_classification.py"
        
        CS[Clustering Services]:::classification
        click CS "https://github.com/jimmc414/document_intelligence/blob/main/cluster_documents.py"
        
        SimA[Similarity Analysis]:::classification
        click SimA "https://github.com/jimmc414/document_intelligence/blob/main/document_similarity.py"
    end

    subgraph Integration
        ESI[External System Integration]:::integration
        AAS[Account Assignment System]:::integration
    end

    %% Connections
    PDF & Email & Audio --> FMS
    FMS --> TE
    TE --> PDF_P & OCR & Audio_P & Email_P
    PDF_P & OCR & Audio_P & Email_P --> NER & KV & SA & TS & TM
    NER & KV & SA & TS & TM --> DC
    DC --> CS & SimA
    CS & SimA --> ESI
    ESI --> AAS

    %% Styles
    classDef input fill:#a8d1f0
    classDef processing fill:#90EE90
    classDef analysis fill:#FFB347
    classDef classification fill:#DDA0DD
    classDef integration fill:#D3D3D3

    %% Legend
    subgraph Legend
        L1[Input Components]:::input
        L2[Processing Services]:::processing
        L3[Analysis Services]:::analysis
        L4[Classification Services]:::classification
        L5[Integration Components]:::integration
    end
```
