# zsolt-varju-serco/open-semantic-search architecture

> Architecture diagram of the GitHub repository zsolt-varju-serco/open-semantic-search, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zsolt-varju-serco/open-semantic-search
- Repository: https://github.com/zsolt-varju-serco/open-semantic-search

## Overview

##### Main Components
- Search Server/Engine – The core of Open Semantic Search. This includes the underlying search engine (primarily built around Apache Solr, as evidenced by the deb package in src/solr.deb and related Docker configurations).
- ETL Framework – The document processing pipeline which handles tasks like crawling, text extraction (using Tika), OCR, named entity recognition (using spaCy and other tools), and even integration of external services. Look at the src/open-semantic-etl module and its associated requirements.
- User Interfaces – This includes the web-based search UI (accessible via docker-compose and the web interface on port 8080) and any additional search apps or desktop search interfaces (the src/open-semantic-desktop-search module and its Vagrantfile and build scripts point to a desktop appliance).
- Connectors – The system is modular and extensible, with connectors for files, databases, RSS, web scraping, etc., as indicated in the extensive docs/connector directory.
- Documentation & Administration – Modules and documentation for admin interfaces, configurations (read docs/doc/admin/*), and configuration of components.
- Integration & Test Environment – Docker configurations for running integration tests (docker-compose.etl.test.yml, docker-compose.test.yml, along with test scripts and Dockerfile-test). Also, end-to-end tests using Playwright and Jest indicate the presence of an automated testing layer.
- External Dependencies – Submodules and external services such as Apache Tika, Apache Solr, and third-party Python libraries (installed via the requirements.txt in the ETL module).

##### Relationships and Interactions
- The ETL framework fetches and processes documents, then pushes enriched data to the search engine index.
- The search engine (Solr) serves queries from the front-end UIs.
- There is clear separation between build/packaging (deb packages, Docker images) and runtime services.
- Integration tests and CI/CD pipelines (through Docker and scripts) indicate that components are deployed together in a containerized setup.

##### Architectural Patterns & Technologies
- Modularity and microservices – Each major function (ETL, search, UI) is encapsulated in its own directory and is built as a separate package/service.
- Containerization – With multiple Dockerfiles (Dockerfile, Dockerfile-cron, Dockerfile-test) and docker-compose files, the architecture is oriented toward containerized deployment.
- Use of Git submodules – Components (like Open Semantic ETL) are managed in separate repositories but integrated via submodules, reinforcing modularity.
- Integration and end-to-end testing – The presence of separate testing configurations shows a commitment to robust integration (via docker-compose.etl.test.yml) and browser automation for UI validation.

##### Additional Recommendations
- Mention testing components explicitly, showing how integration tests and E2E tests are orchestrated in a containerized environment.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Documentation & Administration"
        D["Documentation & Administration"]:::docs
    end

    subgraph "Core Processing"
        direction TB
        C["Connectors"]:::connectors
        X["External Dependencies"]:::external
        E["ETL Framework"]:::etl
        S["Search Engine (Solr)"]:::search
    end

    subgraph "User Interfaces"
        direction TB
        DS["Desktop Search UI"]:::ui
        WS["Web-based Search/App UI"]:::ui
    end

    DC["Containerization & CI/CD Pipeline"]:::deploy
    IEnv["Integration & Test Environment"]:::integration

    %% Data Flow Relationships
    C -->|"dataSourcing"| E
    X -->|"enrichData"| E
    E -->|"indexing"| S
    S -->|"queryResponses"| DS
    S -->|"queryResponses"| WS

    %% Documentation relationships
    D -->|"configures"| E
    D -->|"configures"| S
    D -->|"guidelines"| C
    D -->|"docs"| DS
    D -->|"docs"| WS

    %% Deployment Relationships
    DC -->|"deploys"| E
    DC -->|"deploys"| S
    DC -->|"deploys"| DS
    DC -->|"deploys"| WS

    %% Testing Relationships
    IEnv -->|"runsTests"| DC

    %% Click Events for Component Mapping
    click S "https://github.com/zsolt-varju-serco/open-semantic-search/blob/master/src/solr.deb"
    click S "https://github.com/zsolt-varju-serco/open-semantic-search/tree/master/Dockerfile"
    click S "https://github.com/zsolt-varju-serco/open-semantic-search/blob/master/docker-compose.yml"

    click E "https://github.com/zsolt-varju-serco/open-semantic-search/tree/master/src/open-semantic-etl"
    click E "https://github.com/zsolt-varju-serco/open-semantic-search/tree/master/docs/etl"

    click DS "https://github.com/zsolt-varju-serco/open-semantic-search/tree/master/src/open-semantic-desktop-search"
    click DS "https://github.com/zsolt-varju-serco/open-semantic-search/tree/master/docs/doc/desktop_search"

    click WS "https://github.com/zsolt-varju-serco/open-semantic-search/tree/master/src/open-semantic-search-apps"
    click WS "https://github.com/zsolt-varju-serco/open-semantic-search/tree/master/docs/solr-php-ui"

    click C "https://github.com/zsolt-varju-serco/open-semantic-search/tree/master/docs/connector"

    click D "https://github.com/zsolt-varju-serco/open-semantic-search/tree/master/docs/doc/admin"

    click IEnv "https://github.com/zsolt-varju-serco/open-semantic-search/tree/master/Dockerfile-test"
    click IEnv "https://github.com/zsolt-varju-serco/open-semantic-search/blob/master/docker-compose.etl.test.yml"
    click IEnv "https://github.com/zsolt-varju-serco/open-semantic-search/blob/master/docker-compose.test.yml"
    click IEnv "https://github.com/zsolt-varju-serco/open-semantic-search/tree/master/test"

    click X "https://github.com/zsolt-varju-serco/open-semantic-search/tree/master/src/tika-python"
    click X "https://github.com/zsolt-varju-serco/open-semantic-search/blob/master/src/tika-server.deb"
    click X "https://github.com/zsolt-varju-serco/open-semantic-search/blob/master/src/spacy-services.deb"

    %% Styles
    classDef etl fill:#F0D9FF,stroke:#000,stroke-width:2px
    classDef search fill:#C0F0C0,stroke:#000,stroke-width:2px
    classDef ui fill:#F0E68C,stroke:#000,stroke-width:2px
    classDef connectors fill:#ADD8E6,stroke:#000,stroke-width:2px
    classDef docs fill:#FFB6C1,stroke:#000,stroke-width:2px
    classDef integration fill:#E6E6FA,stroke:#000,stroke-width:2px
    classDef external fill:#FFE4B5,stroke:#000,stroke-width:2px
    classDef deploy fill:#D3D3D3,stroke:#000,stroke-width:2px
```
