How cognitedata/neat works
Main Components
• Client Integration & Session Management:
• “NeatSession” component (found under cognite/neat/_session and its engine submodule) as the primary orchestrator. This component is instantiated by users (typically via a Jupyter Notebook) after creating a CogniteClient.
• “client” wrapper that manages connectivity with external APIs (e.g., Cognite Data Fusion) and is important to enable ingestion of models/graphs.
• Graph Processing Pipeline:
• A “Graph Module” that comprises subcomponents like extractors (e.g., _graph/extractors for handling RDF files, CDF-export specific extractors, and others), loaders (for loading graphs from different sources and formats), queries, and transformers. Emphasize how raw input data is extracted, transformed, and then converted into the target semantic representation.
• Clearly the examples and test cases in the _graph/examples folder, which help validate and illustrate the processing flow.
• Rules Engine:
• Issue Handling:
• Configuration & Utilities:
• Testing & Integration:
• Testing suite (under tests/) as a separate component, showing the integration and unit tests that validate the behavior of the various modules, ensuring that all the layers (session, graph, rules, etc.) have robust verification.
Relationships and Data Flow
• The Cognite client is created.
• A NeatSession is instantiated, which then coordinates activities.
• Next, illustrate how data flows from external sources (e.g., Excel files, RDF files, and other formats captured in the “docs”, “scripts”, and “tests” directories) into the graph extraction and transformation modules.
• Interaction between the graph pipeline and the rules engine, where graph elements may be enriched or validated against the rules.
• Demonstrate the final data flow toward external systems like Cognite Data Fusion (or possibly other production endpoints).
Architectural Patterns and Design Principles
• Modular structure; each subcomponent (client, session, graph, rules, issues) is encapsulated in its own module so that responsibilities are clearly separated.
• A pipeline (ETL-like flow) for transforming raw models and graphs into standardized knowledge graphs.
• Make a note of extensibility points (e.g., additional extractors, importers, or transformers) which allow for future enhancements.
• Use of configuration files (e.g., pyproject.toml, cdf.toml) and testing configurations (_pre-commit-config.yaml, codecov.yml) as elements that support development and deployment best practices.
Technology and Frameworks
• That this is implemented in Python.
• Use of type hints (py.typed), linting (Ruff, Black, Mypy) as part of the development workflow.
• Mention that the notebook-based UI and orchestration (NeatSession) is key for user interactions.
• Point out that integration with Cognite Data Fusion is the primary external dependency.