# cognitedata/neat architecture

> Architecture diagram of the GitHub repository cognitedata/neat, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cognitedata/neat
- Repository: https://github.com/cognitedata/neat

## Overview

##### Main Components
- Client Integration & Session Management:
- “NeatSession” component (found under cognite/neat/_session and its engine submodule) as the primary orchestrator. This component is instantiated by users (typically via a Jupyter Notebook) after creating a CogniteClient.
- “client” wrapper that manages connectivity with external APIs (e.g., Cognite Data Fusion) and is important to enable ingestion of models/graphs.

- Graph Processing Pipeline:
- A “Graph Module” that comprises subcomponents like extractors (e.g., _graph/extractors for handling RDF files, CDF-export specific extractors, and others), loaders (for loading graphs from different sources and formats), queries, and transformers. Emphasize how raw input data is extracted, transformed, and then converted into the target semantic representation.
- Clearly the examples and test cases in the _graph/examples folder, which help validate and illustrate the processing flow.

- Rules Engine:

- Issue Handling:

- Configuration & Utilities:

- Testing & Integration:
- Testing suite (under tests/) as a separate component, showing the integration and unit tests that validate the behavior of the various modules, ensuring that all the layers (session, graph, rules, etc.) have robust verification.

##### Relationships and Data Flow
- The Cognite client is created.
- A NeatSession is instantiated, which then coordinates activities.

- Next, illustrate how data flows from external sources (e.g., Excel files, RDF files, and other formats captured in the “docs”, “scripts”, and “tests” directories) into the graph extraction and transformation modules.

- Interaction between the graph pipeline and the rules engine, where graph elements may be enriched or validated against the rules.

- Demonstrate the final data flow toward external systems like Cognite Data Fusion (or possibly other production endpoints).

##### Architectural Patterns and Design Principles
- Modular structure; each subcomponent (client, session, graph, rules, issues) is encapsulated in its own module so that responsibilities are clearly separated.
- A pipeline (ETL-like flow) for transforming raw models and graphs into standardized knowledge graphs.
- Make a note of extensibility points (e.g., additional extractors, importers, or transformers) which allow for future enhancements.
- Use of configuration files (e.g., pyproject.toml, cdf.toml) and testing configurations (_pre-commit-config.yaml, codecov.yml) as elements that support development and deployment best practices.

##### Technology and Frameworks
- That this is implemented in Python.
- Use of type hints (py.typed), linting (Ruff, Black, Mypy) as part of the development workflow.
- Mention that the notebook-based UI and orchestration (NeatSession) is key for user interactions.
- Point out that integration with Cognite Data Fusion is the primary external dependency.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Global nodes
    NotebookUI["Notebook UI"]:::external

    %% Client Integration & Session Management
    subgraph "Client Integration & Session Management"
        ExternalAPIClient["External API Client"]:::core
        NeatSession["NeatSession"]:::core
    end

    %% Graph Processing Pipeline
    subgraph "Graph Processing Pipeline"
        GraphExtractors["Graph Extractors"]:::core
        GraphTransformers["Graph Transformers"]:::core
        GraphLoaders["Graph Loaders"]:::core
        GraphQueries["Graph Queries"]:::core
        GraphExamples["Graph Examples"]:::core
    end

    %% Rules Engine
    subgraph "Rules Engine"
        RulesImporters["Rules Importers"]:::core
        RulesTransformers["Rules Transformers"]:::core
        RulesExporters["Rules Exporters"]:::core
        RulesModels["Rules Models"]:::core
    end

    %% Other Supporting Components
    IssueHandling["Issue Handling"]:::core

    subgraph "Configuration & Utilities"
        Configuration["Configuration"]:::utility
        Utilities["Utilities"]:::utility
    end

    Testing["Testing & Integration"]:::test
    CogniteDF["Cognite Data Fusion"]:::external

    %% Main Flow Connections
    NotebookUI -->|"create_client"| ExternalAPIClient
    ExternalAPIClient -->|"start_session"| NeatSession
    NeatSession -->|"initiate_extraction"| GraphExtractors
    GraphExtractors -->|"transform_data"| GraphTransformers
    GraphTransformers -->|"load_graph"| GraphLoaders
    GraphLoaders -->|"query_graph"| GraphQueries
    GraphQueries -->|"apply_rules"| RulesImporters
    RulesImporters -->|"process_rules"| RulesTransformers
    RulesTransformers -->|"export_rules"| RulesExporters
    RulesExporters -->|"final_ingestion"| CogniteDF

    %% Auxiliary flow within Graph Pipeline
    GraphExtractors -->|"example_validation"| GraphExamples

    %% Issue Handling Connection
    NeatSession -.->|"log_issues"| IssueHandling

    %% Supporting connections for configuration and testing
    Configuration -.->|"support_config"| NeatSession
    Utilities -.->|"assist_modules"| GraphTransformers
    Testing -.->|"validate_pipeline"| GraphExtractors

    %% Click Events
    click NeatSession "https://github.com/cognitedata/neat/tree/main/cognite/neat/_session/"
    click ExternalAPIClient "https://github.com/cognitedata/neat/tree/main/cognite/neat/_client/"
    click GraphExtractors "https://github.com/cognitedata/neat/tree/main/cognite/neat/_graph/extractors/"
    click GraphLoaders "https://github.com/cognitedata/neat/tree/main/cognite/neat/_graph/loaders/"
    click GraphQueries "https://github.com/cognitedata/neat/tree/main/cognite/neat/_graph/queries/"
    click GraphTransformers "https://github.com/cognitedata/neat/tree/main/cognite/neat/_graph/transformers/"
    click GraphExamples "https://github.com/cognitedata/neat/tree/main/cognite/neat/_graph/examples/"
    click RulesImporters "https://github.com/cognitedata/neat/tree/main/cognite/neat/_rules/importers/"
    click RulesExporters "https://github.com/cognitedata/neat/tree/main/cognite/neat/_rules/exporters/"
    click RulesTransformers "https://github.com/cognitedata/neat/tree/main/cognite/neat/_rules/transformers/"
    click RulesModels "https://github.com/cognitedata/neat/tree/main/cognite/neat/_rules/models/"
    click IssueHandling "https://github.com/cognitedata/neat/tree/main/cognite/neat/_issues/"
    click Configuration "https://github.com/cognitedata/neat/blob/main/cognite/neat/_config.py"
    click Utilities "https://github.com/cognitedata/neat/tree/main/cognite/neat/_utils/"
    click Testing "https://github.com/cognitedata/neat/tree/main/tests/"

    %% Styles
    classDef core fill:#b3cde0,stroke:#6497b1,stroke-width:2px;
    classDef external fill:#fbb4ae,stroke:#e41a1c,stroke-width:2px;
    classDef utility fill:#ccebc5,stroke:#4daf4a,stroke-width:2px;
    classDef test fill:#decbe4,stroke:#b15928,stroke-width:2px;
```
