# harmonydata/harmony architecture

> Architecture diagram of the GitHub repository harmonydata/harmony, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/harmonydata/harmony
- Repository: https://github.com/harmonydata/harmony

## Overview

##### Main Components Identification

The system should be divided into these primary components:

a) Input Processing Layer
- PDF Parser
- Excel Parser
- Text Parser
- File Type Detection

b) Core Processing Layer
- Matching Engine
- Vectorization Service
- Clustering System
- Negation Handler

c) Schema Layer
- Instrument Schemas
- Question Schemas
- Request/Response Models

d) Utility Services
- Model Management
- File Helpers
- Instrument Helpers

##### Component Relationships

**[Input Files (PDF/Excel/Text)] → [Parsing Layer] → [Schema Layer] → [Core Processing Layer] → [Output Results]**

##### Specific Implementation Details

a) Parser Components:
- Tika integration for PDF processing
- Excel parsing capabilities
- Text parsing direct input

b) Matching System:
- Default matcher
- WMD matcher
- Clustering functionality
- Vector generation system

c) External Dependencies:
- Apache Tika Server
- Sentence Transformers
- spaCy Models
- Optional OpenAI integration

##### Key Architectural Patterns to Highlight

- Modular design with clear separation of concerns
- Plugin architecture for different matchers
- Service-based approach for core functionalities
- Schema-driven data validation

##### Implementation Details

**[File Input] → [Parser Selection] → [Text Extraction] → [Vector Generation] → [Similarity Matching] → [Results]**

With parallel flows for:
- Model management
- Schema validation
- Error handling

##### Technical Stack Indicators

- Python 3.8+
- spaCy
- Sentence Transformers
- Apache Tika
- Docker support

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Input
        Files[("Input Files<br/>(PDF/Excel/Text)")]:::input
    end

    subgraph "Parsing Layer"
        FTD["File Type Detection"]:::parser
        PDF["PDF Parser"]:::parser
        Excel["Excel Parser"]:::parser
        Text["Text Parser"]:::parser
        Tika["Tika Integration"]:::external
    end

    subgraph "Schema Layer"
        IS["Instrument Schemas"]:::schema
        QS["Question Schemas"]:::schema
        ReqM["Request Models"]:::schema
        ResM["Response Models"]:::schema
        TV["Text Vector Schema"]:::schema
    end

    subgraph "Core Processing Layer"
        ME["Matching Engine"]:::core
        DM["Default Matcher"]:::core
        WM["WMD Matcher"]:::core
        CS["Clustering System"]:::core
        NH["Negation Handler"]:::core
        VG["Vector Generation"]:::core
        CG["Crosswalk Generation"]:::core
    end

    subgraph "External Services"
        Tika_Server["Apache Tika Server"]:::external
        ST["Sentence Transformers"]:::external
        Spacy["spaCy Models"]:::external
        OpenAI["OpenAI Integration"]:::external_optional
    end

    subgraph "Utility Services"
        MM["Model Management"]:::utility
        FH["File Helpers"]:::utility
        IH["Instrument Helpers"]:::utility
    end

    Files --> FTD
    FTD --> PDF & Excel & Text
    PDF --> Tika
    Tika --> Tika_Server

    PDF & Excel & Text --> IS
    IS & QS --> ME
    ReqM & ResM --> ME
    
    ME --> DM & WM
    DM & WM --> CS
    CS --> NH
    NH --> VG
    VG --> CG
    
    ST & Spacy --> VG
    OpenAI -.-> VG
    
    MM --> ST & Spacy
    FH --> Files
    IH --> IS

    %% Styling
    classDef input fill:#90EE90
    classDef parser fill:#ADD8E6
    classDef schema fill:#FFE4B5
    classDef core fill:#4682B4
    classDef external fill:#FFA07A
    classDef external_optional fill:#FFA07A,stroke-dasharray: 5 5
    classDef utility fill:#D3D3D3

    %% Click events
    click FTD "https://github.com/harmonydata/harmony/blob/main/src/harmony/schemas/enums/file_types.py"
    click PDF "https://github.com/harmonydata/harmony/blob/main/src/harmony/parsing/pdf_parser.py"
    click Excel "https://github.com/harmonydata/harmony/blob/main/src/harmony/parsing/excel_parser.py"
    click Text "https://github.com/harmonydata/harmony/blob/main/src/harmony/parsing/text_parser.py"
    click Tika "https://github.com/harmonydata/harmony/blob/main/src/harmony/parsing/util/tika_wrapper.py"
    click IS "https://github.com/harmonydata/harmony/blob/main/src/harmony/schemas/catalogue_instrument.py"
    click QS "https://github.com/harmonydata/harmony/blob/main/src/harmony/schemas/catalogue_question.py"
    click ReqM "https://github.com/harmonydata/harmony/tree/main/src/harmony/schemas/requests/"
    click ResM "https://github.com/harmonydata/harmony/tree/main/src/harmony/schemas/responses/"
    click TV "https://github.com/harmonydata/harmony/blob/main/src/harmony/schemas/text_vector.py"
    click ME "https://github.com/harmonydata/harmony/blob/main/src/harmony/matching/matcher.py"
    click DM "https://github.com/harmonydata/harmony/blob/main/src/harmony/matching/default_matcher.py"
    click WM "https://github.com/harmonydata/harmony/blob/main/src/harmony/matching/wmd_matcher.py"
    click CS "https://github.com/harmonydata/harmony/blob/main/src/harmony/matching/cluster.py"
    click NH "https://github.com/harmonydata/harmony/blob/main/src/harmony/matching/negator.py"
    click CG "https://github.com/harmonydata/harmony/blob/main/src/harmony/matching/generate_crosswalk_table.py"
    click MM "https://github.com/harmonydata/harmony/blob/main/src/harmony/util/model_downloader.py"
    click FH "https://github.com/harmonydata/harmony/blob/main/src/harmony/util/file_helper.py"
    click IH "https://github.com/harmonydata/harmony/blob/main/src/harmony/util/instrument_helper.py"
```
