How geneialco/geneial-langmanus-transform works
Project type and purpose
• This is primarily a command‐line tool (CLI) for transforming input data (e.g., TOPMED YAML) into target formats (LinkML, Phenopackets) via an AI‐agent workflow. It also bundles an optional HTTP API (server.py + src/api/app.py) and ships with a Docker setup for containerized runs.
File‐structure analysis and high‐level architecture
• Entry points
• src/cli.py – CLI; parses arguments and invokes the core workflow.
• server.py (+ src/api/app.py) – HTTP server exposing the same workflow as REST endpoints.
• Core engine
• src/service/workflow_service.py & src/workflow.py – orchestrate the full data‐transformation pipeline: ingest, agent loop, validation, output.
• Modular subsystems
• Agents (src/agents) – controller logic for orchestrating LLM calls & retries.
• LLM clients (src/llms) – abstractions over underlying LLM providers (e.g., LiteLLM, OpenAI).
• Prompts (src/prompts) – prompt templates and policies for each agent role.
• Tools (src/tools) – pluggable capabilities (browser automation, file I/O, web crawling, Python REPL, search).
• Crawler (src/crawler) – specialized data‐gathering logic (article extraction, readability, Jina client).
• Graph builder (src/graph) – intermediate representation builder (nodes, types, builder).
• Config (src/config) – loader for environment, YAML, tracing settings, agent/tool registration.
• External dependencies
• LinkML CLI (linkml-validate) – invoked for schema validation.
• Docker / docker-compose – containerization.
• CI/CD workflows in GitHub Actions.
Key architectural patterns
• Pipeline/Orchestration – the workflow service drives a sequence of steps (generation → validation → retry).
• Plugin/strategy – tools and agents are loosely coupled via interfaces.
• Hexagonal/Clean architecture – core logic (workflow + agents) is isolated from I/O (CLI, HTTP, file, network).
Components
a. Entry Points
• CLI module (src/cli.py)
• HTTP API (server.py → src/api/app.py)
b. Core Orchestrator
• WorkflowService / workflow.py
c. Configuration Layer
• src/config (loaders, env, tracing)
d. Agent Layer
• src/agents (agent controller, retry logic)
• Prompt templates (src/prompts)
e. Execution Tools
• src/tools (bash, browser, file management, crawl, search, python_repl)
f. Data Gatherer
• src/crawler (article, readability, Jina)
g. Graph Construction
• src/graph (nodes, types, builder)
h. LLM Provider
• src/llms (litellm_v2, llm abstraction) → external LLM API/service
i. Validation
• External linkml-validate tool
j. Persistence & I/O
• File system (input YAML, output YAML/CSV)
• Static assets, examples
Relationships & data flows
• User → (CLI or HTTP) → WorkflowService
• WorkflowService → reads Config
• WorkflowService → dispatches Agents
• Agents → load Prompt templates → send requests to LLM Provider
• Agents → invoke Tools and Crawler as needed → receive tool output
• Agents → assemble an intermediate graph via Graph Builder
• WorkflowService → writes output files (YAML/CSV)
• WorkflowService (if --validate) → calls external linkml-validate → on failure loops back through the agent