# populationgenomics/metamist architecture

> Architecture diagram of the GitHub repository populationgenomics/metamist, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/populationgenomics/metamist
- Repository: https://github.com/populationgenomics/metamist

## Overview

##### Project Type & Purpose
- This is a full-stack application (“Metamist”) designed to manage and query -omics metadata.
- It consists of a robust MariaDB database for data storage, a Python web API for access control and query handling (using GraphQL and HTTP), and clients (a React web application and an auto-generated Python client) to interact with the system.
- The README highlights key features such as project-based data organization and permission management via Google Groups (with caching) for performance.

##### Analyzing the File Structure
- The top-level directories hint at a multi-component architecture:
- “api”: Contains the Python web server with subdirectories for GraphQL (schema, mutations, loaders, filters) and HTTP routes.
- “db”: Encompasses database related logic including connections, deployment (Dockerfiles, liquibase configurations), and Python modules for table definitions and business logic layers.
- “models”: Defines core data models, enums, and shared utilities that are used both by the API and by database interactions.
- “metamist” and “metamist_infrastructure”: Provide additional application-specific functionalities and infrastructure code.
- “etl”: Contains modules for data extraction, loading, and notifications – representing external data manipulation components.
- “web”: Houses a React (TypeScript) client that consumes the GraphQL API and the generated Typescript client.
- Additionally, build and deployment configurations, tests, and scripts are included to support continuous integration and deployment, lending insight into how the overall system is maintained.

##### Key Components
a. Database Layer
- A system-versioned MariaDB database (with schema as described in the README and illustrated in the provided schema image).
- Supporting modules from “db/python/tables” and “db/python/layers” show the logic to interface with MariaDB and BigQuery.
b. API Layer
- The Python Web API serves as the central engine. It is divided into:
- GraphQL endpoints (in “api/graphql” for queries, mutations, and schema definitions).
- HTTP routes (in “api/routes”) covering REST/OpenAPI interfaces.
- Internal modules handling permissions, caching of Google Group memberships (for access control), and frequently used queries.
- Use of Python client libraries (wrapped via OpenAPI) which allow external programs to interact with the API.
c. Front-End Client Layer
- The “web” directory contains the React (TypeScript) application that interacts with the API via generated Typescript clients and via GraphQL.
d. Auxiliary Components
- ETL processes (in “etl”) for data extraction, loading, and sending notifications.
- Infrastructure and deployment components (scripts, Dockerfiles, configuration files) that support deployment and operational maintenance.

##### Relationships and Interactions
- The React “web” client and the autogenerated Python client communicate with the Python Web API.
- The API accesses the MariaDB database (and BigQuery components where relevant) for CRUD operations.
- ETL processes may either feed data directly into the database or interact with the API.
- Internal caching of Google Groups in the API enforces access control.
- Clearly label communication types (e.g., “GraphQL queries”, “HTTP REST calls”, “Database CRUD operations”).

##### Architectural Patterns and Design Principles
- Modular and layered architecture: separate concerns across database, business logic (in API), and presentation (React client).
- Separation of API concerns: distinct handling for GraphQL requests, REST/OAS endpoints, and internal data models.
- Use of auto-generated client libraries to enforce consistency between API contracts and client usage.
- Caching (for Google Group membership) as a critical design decision made to optimize performance.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Client Layer
    subgraph "Client Layer"
        React["React Front-End Client"]:::client
        PythonClient["Python Client Library"]:::client
        OpenAPITemplates["OpenAPI Templates"]:::client
        RegenerateScript["Regeneration Script"]:::client
    end

    %% API Layer
    subgraph "API Layer"
        GraphQL["GraphQL Endpoints"]:::api
        HTTP["HTTP REST Endpoints"]:::api
        ServerSettings["Server & Settings"]:::api
        Utility["Utility Modules"]:::api
    end

    %% Database Layer
    subgraph "Database Layer"
        DB["MariaDB Database"]:::db
        Tables["Tables Module"]:::db
        DBLogic["Database Logic"]:::db
    end

    %% ETL Processes
    subgraph "ETL Processes"
        ETL["ETL Services"]:::etl
    end

    %% Infrastructure & Deployment
    subgraph "Infrastructure & Deployment"
        MetamistApp["Metamist Application"]:::infra
        InfraCode["Infrastructure Code"]:::infra
    end

    %% External Dependencies
    GGroups["Google Groups"]:::external

    %% Connections: Clients to API Layer
    React -->|"GraphQL_queries"| GraphQL
    PythonClient -->|"REST_calls"| HTTP
    %% Also connect Python Client internal components
    PythonClient --> OpenAPITemplates
    PythonClient --> RegenerateScript

    %% Connections: API Layer to Database Layer
    GraphQL -->|"CRUD_operations"| DB
    HTTP -->|"CRUD_operations"| DB
    ServerSettings -->|"access"| Tables
    ServerSettings -->|"access"| DBLogic
    Utility -->|"caches"| GGroups

    %% Internal Database Layer composition
    Tables --> DB
    DBLogic --> DB

    %% ETL Processes interactions
    ETL ---|"Data_Loading"| DB
    ETL ---|"Notifications"| GraphQL

    %% Infrastructure interactions
    MetamistApp ---|"Deployment_Config"| ServerSettings
    InfraCode ---|"Deployment_Config"| DB

    %% Style definitions
    classDef client fill:#add8e6,stroke:#000,stroke-width:1px;
    classDef api fill:#ffcccb,stroke:#000,stroke-width:1px;
    classDef db fill:#c1f0c1,stroke:#000,stroke-width:1px;
    classDef etl fill:#fde047,stroke:#000,stroke-width:1px,stroke-dasharray: 5 5;
    classDef infra fill:#d3d3d3,stroke:#000,stroke-width:1px,stroke-dasharray: 3 3;
    classDef external fill:#ffeb99,stroke:#000,stroke-width:1px;

    %% Click Events
    click Tables "https://github.com/populationgenomics/metamist/tree/dev/db/python/tables/"
    click DBLogic "https://github.com/populationgenomics/metamist/tree/dev/db/python/layers/"
    click GraphQL "https://github.com/populationgenomics/metamist/blob/dev/api/graphql/schema.py"
    click HTTP "https://github.com/populationgenomics/metamist/blob/dev/api/routes/analysis.py"
    click ServerSettings "https://github.com/populationgenomics/metamist/blob/dev/api/server.py"
    click Utility "https://github.com/populationgenomics/metamist/blob/dev/api/utils/gcp.py"
    click OpenAPITemplates "https://github.com/populationgenomics/metamist/blob/dev/openapi-templates/api.mustache"
    click RegenerateScript "https://github.com/populationgenomics/metamist/blob/dev/regenerate_api.py"
    click React "https://github.com/populationgenomics/metamist/blob/dev/web/package.json"
    click ETL "https://github.com/populationgenomics/metamist/tree/dev/etl/extract/"
    click MetamistApp "https://github.com/populationgenomics/metamist/tree/dev/metamist/"
    click InfraCode "https://github.com/populationgenomics/metamist/tree/dev/metamist_infrastructure/"
```
