# geneialco/geneial-langmanus-transform architecture

> Architecture diagram of the GitHub repository geneialco/geneial-langmanus-transform, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/geneialco/geneial-langmanus-transform
- Repository: https://github.com/geneialco/geneial-langmanus-transform

## Overview

##### Project type and purpose
- This is primarily a **command‐line tool** (CLI) for transforming input data (e.g., TOPMED YAML) into target formats (LinkML, Phenopackets) via an AI‐agent workflow. It also bundles an **optional HTTP API** (`server.py` + `src/api/app.py`) and ships with a Docker setup for containerized runs.

##### File‐structure analysis and high‐level architecture
- **Entry points**
- `src/cli.py` – CLI; parses arguments and invokes the core workflow.
- `server.py` (+ `src/api/app.py`) – HTTP server exposing the same workflow as REST endpoints.
- **Core engine**
- `src/service/workflow_service.py` & `src/workflow.py` – orchestrate the full data‐transformation pipeline: ingest, agent loop, validation, output.
- **Modular subsystems**
- **Agents** (`src/agents`) – controller logic for orchestrating LLM calls & retries.
- **LLM clients** (`src/llms`) – abstractions over underlying LLM providers (e.g., LiteLLM, OpenAI).
- **Prompts** (`src/prompts`) – prompt templates and policies for each agent role.
- **Tools** (`src/tools`) – pluggable capabilities (browser automation, file I/O, web crawling, Python REPL, search).
- **Crawler** (`src/crawler`) – specialized data‐gathering logic (article extraction, readability, Jina client).
- **Graph builder** (`src/graph`) – intermediate representation builder (nodes, types, builder).
- **Config** (`src/config`) – loader for environment, YAML, tracing settings, agent/tool registration.
- **External dependencies**
- **LinkML** CLI (`linkml-validate`) – invoked for schema validation.
- **Docker** / `docker-compose` – containerization.
- **CI/CD workflows** in GitHub Actions.

##### Key architectural patterns
- **Pipeline/Orchestration** – the workflow service drives a sequence of steps (generation → validation → retry).
- **Plugin/strategy** – tools and agents are loosely coupled via interfaces.
- **Hexagonal/Clean architecture** – core logic (workflow + agents) is isolated from I/O (CLI, HTTP, file, network).

##### Components
a. **Entry Points**
- CLI module (`src/cli.py`)
- HTTP API (`server.py` → `src/api/app.py`)
b. **Core Orchestrator**
- `WorkflowService` / `workflow.py`
c. **Configuration Layer**
- `src/config` (loaders, env, tracing)
d. **Agent Layer**
- `src/agents` (agent controller, retry logic)
- **Prompt templates** (`src/prompts`)
e. **Execution Tools**
- `src/tools` (bash, browser, file management, crawl, search, python_repl)
f. **Data Gatherer**
- `src/crawler` (article, readability, Jina)
g. **Graph Construction**
- `src/graph` (nodes, types, builder)
h. **LLM Provider**
- `src/llms` (litellm_v2, llm abstraction) → external LLM API/service
i. **Validation**
- External `linkml-validate` tool
j. **Persistence & I/O**
- File system (input YAML, output YAML/CSV)
- Static assets, examples

##### Relationships & data flows
- **User** → (CLI or HTTP) → **WorkflowService**
- **WorkflowService** → reads **Config**
- **WorkflowService** → dispatches **Agents**
- **Agents** → load **Prompt templates** → send requests to **LLM Provider**
- **Agents** → invoke **Tools** and **Crawler** as needed → receive tool output
- **Agents** → assemble an **intermediate graph** via **Graph Builder**
- **WorkflowService** → writes output files (YAML/CSV)
- **WorkflowService** (if `--validate`) → calls external **linkml-validate** → on failure loops back through the agent

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Entry Layer
    subgraph "Entry Layer"
        direction TB
        CLI["CLI (src/cli.py)"]:::entry
        HTTP["HTTP API Server\n(server.py → src/api/app.py)"]:::entry
    end

    %% Core Orchestration
    subgraph "Core Orchestration"
        direction TB
        WorkflowSvc["WorkflowService\n(src/service/workflow_service.py)"]:::core
        TopWorkflow["Workflow\n(src/workflow.py)"]:::core
    end

    %% Configuration
    subgraph "Configuration Layer"
        direction TB
        Config["Config\n(src/config/)"]:::config
    end

    %% Agent Layer
    subgraph "Agent Layer"
        direction TB
        Agents["Agents Controller\n(src/agents/agents.py)"]:::agent
        Prompts["Prompt Templates\n(src/prompts/)"]:::agent
    end

    %% Execution Layer
    subgraph "Execution & Data Processing"
        direction TB
        Tools["Execution Tools\n(src/tools/)"]:::tools
        Crawler["Data Gatherer\n(src/crawler/)"]:::tools
        GraphBuilder["Graph Builder\n(src/graph/)"]:::tools
        LLM["LLM Provider\n(src/llms/)"]:::tools
        FileIO["File Management\n(src/tools/file_management.py)"]:::tools
    end

    %% External Services
    subgraph "External Services"
        direction TB
        LinkML["LinkML Validator\n(linkml-validate)"]:::external
    end

    %% Persistence
    subgraph "Persistence & I/O"
        direction TB
        FS["File System\n(example-topmed.yaml,\nexample-linkml-output.yaml)"]:::external
    end

    %% CI/CD & Docker
    subgraph "DevOps"
        direction TB
        Docker["Docker\n(Dockerfile,\ndocker-compose.yml)"]:::external
        CICD["CI/CD Workflows\n(.github/workflows/)"]:::external
    end

    %% Flows
    CLI -->|"invoke"| WorkflowSvc
    HTTP -->|"REST call"| WorkflowSvc
    WorkflowSvc -->|"load settings"| Config
    WorkflowSvc -->|"start pipeline"| TopWorkflow
    TopWorkflow -->|"dispatch"| Agents
    Agents -->|"load prompts"| Prompts
    Agents -->|"call"| LLM
    Agents -->|"invoke tools"| Tools
    Agents -->|"use crawler"| Crawler
    Agents -->|"build graph"| GraphBuilder
    TopWorkflow -->|"write outputs"| FileIO
    TopWorkflow -->|"write outputs"| FS
    TopWorkflow -->|"validate?"| LinkML
    LinkML -.->|"failure"| Agents
    LinkML -->|"success"| FS

    %% Click Events
    click CLI "https://github.com/geneialco/geneial-langmanus-transform/blob/main/src/cli.py"
    click HTTP "https://github.com/geneialco/geneial-langmanus-transform/blob/main/server.py"
    click HTTP "https://github.com/geneialco/geneial-langmanus-transform/blob/main/src/api/app.py"
    click WorkflowSvc "https://github.com/geneialco/geneial-langmanus-transform/blob/main/src/service/workflow_service.py"
    click TopWorkflow "https://github.com/geneialco/geneial-langmanus-transform/blob/main/src/workflow.py"
    click Config "https://github.com/geneialco/geneial-langmanus-transform/tree/main/src/config/"
    click Agents "https://github.com/geneialco/geneial-langmanus-transform/blob/main/src/agents/agents.py"
    click Prompts "https://github.com/geneialco/geneial-langmanus-transform/tree/main/src/prompts/"
    click Tools "https://github.com/geneialco/geneial-langmanus-transform/tree/main/src/tools/"
    click Crawler "https://github.com/geneialco/geneial-langmanus-transform/tree/main/src/crawler/"
    click GraphBuilder "https://github.com/geneialco/geneial-langmanus-transform/tree/main/src/graph/"
    click LLM "https://github.com/geneialco/geneial-langmanus-transform/tree/main/src/llms/"
    click FileIO "https://github.com/geneialco/geneial-langmanus-transform/blob/main/src/tools/file_management.py"
    click LinkML "https://github.com/geneialco/geneial-langmanus-transform/tree/main/linkml-validate"
    click FS "https://github.com/geneialco/geneial-langmanus-transform/blob/main/example-topmed.yaml"
    click FS "https://github.com/geneialco/geneial-langmanus-transform/blob/main/example-linkml-output.yaml"
    click Docker "https://github.com/geneialco/geneial-langmanus-transform/tree/main/Dockerfile"
    click Docker "https://github.com/geneialco/geneial-langmanus-transform/blob/main/docker-compose.yml"
    click CICD "https://github.com/geneialco/geneial-langmanus-transform/tree/main/.github/workflows/"

    %% Styles
    classDef entry fill:#D0E8FF,stroke:#0288D1,color:#000;
    classDef core fill:#E0F7FA,stroke:#0097A7,color:#000;
    classDef config fill:#FFF3E0,stroke:#FB8C00,color:#000;
    classDef agent fill:#FFFDE7,stroke:#FBC02D,color:#000;
    classDef tools fill:#E8F5E9,stroke:#43A047,color:#000;
    classDef external fill:#FBE9E7,stroke:#E64A19,color:#000;
```
