How populationgenomics/metamist works
Project Type & Purpose
• This is a full-stack application (“Metamist”) designed to manage and query -omics metadata.
• It consists of a robust MariaDB database for data storage, a Python web API for access control and query handling (using GraphQL and HTTP), and clients (a React web application and an auto-generated Python client) to interact with the system.
• The README highlights key features such as project-based data organization and permission management via Google Groups (with caching) for performance.
Analyzing the File Structure
• The top-level directories hint at a multi-component architecture:
• “api”: Contains the Python web server with subdirectories for GraphQL (schema, mutations, loaders, filters) and HTTP routes.
• “db”: Encompasses database related logic including connections, deployment (Dockerfiles, liquibase configurations), and Python modules for table definitions and business logic layers.
• “models”: Defines core data models, enums, and shared utilities that are used both by the API and by database interactions.
• “metamist” and “metamist_infrastructure”: Provide additional application-specific functionalities and infrastructure code.
• “etl”: Contains modules for data extraction, loading, and notifications – representing external data manipulation components.
• “web”: Houses a React (TypeScript) client that consumes the GraphQL API and the generated Typescript client.
• Additionally, build and deployment configurations, tests, and scripts are included to support continuous integration and deployment, lending insight into how the overall system is maintained.
Key Components
a. Database Layer
• A system-versioned MariaDB database (with schema as described in the README and illustrated in the provided schema image).
• Supporting modules from “db/python/tables” and “db/python/layers” show the logic to interface with MariaDB and BigQuery.
b. API Layer
• The Python Web API serves as the central engine. It is divided into:
• GraphQL endpoints (in “api/graphql” for queries, mutations, and schema definitions).
• HTTP routes (in “api/routes”) covering REST/OpenAPI interfaces.
• Internal modules handling permissions, caching of Google Group memberships (for access control), and frequently used queries.
• Use of Python client libraries (wrapped via OpenAPI) which allow external programs to interact with the API.
c. Front-End Client Layer
• The “web” directory contains the React (TypeScript) application that interacts with the API via generated Typescript clients and via GraphQL.
d. Auxiliary Components
• ETL processes (in “etl”) for data extraction, loading, and sending notifications.
• Infrastructure and deployment components (scripts, Dockerfiles, configuration files) that support deployment and operational maintenance.
Relationships and Interactions
• The React “web” client and the autogenerated Python client communicate with the Python Web API.
• The API accesses the MariaDB database (and BigQuery components where relevant) for CRUD operations.
• ETL processes may either feed data directly into the database or interact with the API.
• Internal caching of Google Groups in the API enforces access control.
• Clearly label communication types (e.g., “GraphQL queries”, “HTTP REST calls”, “Database CRUD operations”).
Architectural Patterns and Design Principles
• Modular and layered architecture: separate concerns across database, business logic (in API), and presentation (React client).
• Separation of API concerns: distinct handling for GraphQL requests, REST/OAS endpoints, and internal data models.
• Use of auto-generated client libraries to enforce consistency between API contracts and client usage.
• Caching (for Google Group membership) as a critical design decision made to optimize performance.