How mannlabs/ckg works
Project Scope and Purpose
• The CKG is an open‐source Python platform designed for integrating and analyzing clinical proteomics data and mining biomedical databases.
• It combines data ingestion from various biomedical sources, complex graph database build/connectivity, analytics and visualization (including statistical methods such as Kaplan–Meier and WGCNA), and a reporting/dashboard UI.
• The project is research-oriented and serves as both a data integration platform and an analytics/reporting tool.
File Structure and Main Components
• There is a clear modular division:
– The “ckg/analytics_core” package contains the primary analysis algorithms, including modules for analytics (e.g., Kaplan–Meier analysis, WGCNA) and visualization.
– The “ckg/graphdb_builder” module comprises subdirectories for building and ingesting data into the graph database. This includes separate areas for databases, experiments, ontologies, and users – each with configuration files, cypher query templates, and parser implementations.
– The “ckg/graphdb_connector” is responsible for connecting to and performing queries on the graph database.
– The “ckg/report_manager” module provides the web/reporting interface, complete with sub-applications (adminApp, dataUpload, project creation, etc.), assets, templates, and worker processes to manage asynchronous tasks.
– The “ckg/notebooks” directory contains Jupyter notebooks for interactive recipes, demonstration of workflows, and a set of example projects.
– There are configuration directories (e.g., “ckg/config”) and resource files (e.g., Dockerfile, neo4j configurations, etc.) which help in deployment and environment setup.
a. And Label Key Components:
• Data Ingestion & GraphDB Builder:
• Modules: ckg/graphdb_builder (submodules: databases, experiments, ontologies, users).
• Parsers: Ingest and parse data from external biomedical databases.
• Configuration files: YAML templates and cypher queries.
• Graph Database Connector:
• Module: ckg/graphdb_connector.
• Role: Provide an interface (with pre-defined queries) to interact with the graph database.
• Analytics and Visualization:
• Module: ckg/analytics_core.
• Algorithms: Statistical and machine learning techniques (e.g., Kaplan–Meier, WGCNA analysis).
• Viz Sub-module: Provides figures like dendrograms and network graphs.
• Reporting and User Interface:
• Module: ckg/report_manager.
• Sub-applications: Login, data upload, project creation, dashboard home, admin.
• Assets and templates: For the presentation of clinical reports.
• Interactive Development/Exploration:
• Notebooks: Located in ckg/notebooks, provide recipes and demonstrative workflows.
• Infrastructure and Deployment:
• Docker support: Dockerfile for containerized deployment.
• Documentation files and configuration resources (e.g., docs, resources/neo4j_db, uwsgi, nginx configuration) to support system-wide deployment.
• External Dependencies:
• Graph Database Engine (likely Neo4j, suggested by configuration files).
• Third-party biomedical databases (the README lists many, such as UniProt, STRING, Reactome, etc.).
b. Relationships and Data Flow:
• Flow from data parsers and mapping components in the “graphdb_builder” to the Graph Database.
• The GraphDB Connector then acts as a mediator, allowing the Analytics Core and the Report Manager to query the graph data.
• The Analytics Core processes data and creates insights, which can then feed back to the Report Manager.
• The Report Manager interfaces with the end-user, displaying reports and dashboards; meanwhile, the Notebooks provide exploratory and recipe-based access to the data.
c. Architectural Patterns and Key Principles:
• Modularization: The project is clearly split into several modules, each with an independent concern (data ingestion, analytics, reporting).
• Layered Architecture: Separate layers for data access, business logic (analytics), and the presentation (report manager and notebooks).
• Use of configuration-driven design: YAML and configuration files indicate that behavior in data ingestion and report generation can be tuned.
• Containerization: Dockerfile ensures that the project can be deployed in a consistent environment.
d. Relevant Technologies and Frameworks:
• Python as the core language with additional libraries for analytics and visualization.
• Neo4j (inferred) for the graph database.
• Docker for containerized deployments.
• Jupyter Notebooks for interactive analyses.
• Web frameworks (potentially Flask or Dash, given the structure of the report_manager/apps) for the report/user interface.
• YAML for configuration management.