# dataiku/dataiku-contrib architecture

> Architecture diagram of the GitHub repository dataiku/dataiku-contrib, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dataiku/dataiku-contrib
- Repository: https://github.com/dataiku/dataiku-contrib

## Overview

##### Main Components
- Plugins: Each sub-directory (e.g., “airtable”, “anonymizer”, “clear-intermediate-datasets”, “deeplearning-image-cpu”, “dowhy-causal-inference”, “forecastio”, etc.) represents an individual plugin. Although there are many plugins, they’re all based on a common plugin architecture pattern.
- Plugin Components: For each plugin module, identify common sub-components such as:
– Plugin Descriptor (plugin.json) – which defines the plugin’s manifest and metadata.
– Custom Recipes – locations where new functionalities (e.g., scoring, prediction, enrichment, etc.) are implemented (typically under directories like custom-recipes).
– Connectors – components under python-connectors (or their language equivalent) that enable integration with external systems.
– Code Environments and Libraries – directories like code-env, python-lib, and similar folders that define runtime environments and include common/shared code.
– Runnables – directories that provide executable scripts (python-runnables, etc.) supporting custom operations.

##### Relationships and Workflows
- Within each plugin, illustrate the internal separation: how the descriptor defines the interface, while custom recipes and connectors form the implementation modules.
- Integration pattern: DSS (the central engine) acts as the orchestrator, loading, validating, and running the plugins during different stages of a data science workflow.

##### Architectural Patterns and Design Principles
- Plugin-based Design: Clearly highlight the extensibility principle—each plugin is independently developed yet adheres to a common structure that DSS can discover and integrate.
- Separation of Concerns: The design should show the decoupling between presentation elements (if any JavaScript or web-app templates are involved), execution logic (runnables, connectors), and configuration (plugin.json and code-env descriptors).
- Language and Environment Isolation: Since plugins may use different languages (Python, Java, JavaScript), indicate that the architecture supports heterogeneous technology stacks by having isolated environments per plugin.

##### Specific Technologies and Libraries
- Fact that many plugins rely on Python (with defined requirements in code-env/python/spec/requirements.txt or similar), as well as Java and JavaScript components.
- Mention that some plugins include web-app templates and resources (CSS, HTML, JS files), indicating front-end integrations.
- Although each plugin might have its own specific libraries, the recurring presence of standardized directories (like python-connectors, python-lib, custom-recipes) indicates adherence to common Dataiku DSS plugin development practices.

## Diagram (Mermaid)

```mermaid
flowchart TD
    DSS_Core["DSS Core Platform"]:::dss

    %% Airtable Plugin
    subgraph "Airtable Plugin"
        A1["Plugin Descriptor (Manifest)"]:::plugin_desc
        A2["Code Environment"]:::code_env
        A4["Connector"]:::connector
        A5["Code Library"]:::lib
        A1 --> A2
        A1 --> A4
        A1 --> A5
    end
    DSS_Core --> A1

    %% Anonymizer Plugin
    subgraph "Anonymizer Plugin"
        AN1["Plugin Descriptor (Manifest)"]:::plugin_desc
    end
    DSS_Core --> AN1

    %% Clear-Intermediate-Datasets Plugin
    subgraph "Clear-Intermediate-Datasets Plugin"
        CID1["Plugin Descriptor (Manifest)"]:::plugin_desc
        CID2["Custom Recipe"]:::custom_recipe
        CID3["Code Library"]:::lib
        CID4["Runnable"]:::runnable
        CID1 --> CID2
        CID1 --> CID3
        CID1 --> CID4
    end
    DSS_Core --> CID1

    %% Deeplearning-image-cpu Plugin
    subgraph "Deeplearning-image-cpu Plugin"
        DIC1["Plugin Descriptor (Manifest)"]:::plugin_desc
        DIC2["Code Environment"]:::code_env
        DIC3["Custom Recipe"]:::custom_recipe
        DIC4["Code Library: API Designer Utils"]:::lib
        DIC5["Code Library: Constants"]:::lib
        DIC6["Runnable"]:::runnable
        DIC7["Web-App Template"]:::webapp
        DIC1 --> DIC2
        DIC1 --> DIC3
        DIC1 --> DIC4
        DIC1 --> DIC5
        DIC1 --> DIC6
        DIC1 --> DIC7
    end
    DSS_Core --> DIC1

    %% Dowhy-causal-inference Plugin
    subgraph "Dowhy-causal-inference Plugin"
        DCI1["Plugin Descriptor (Manifest)"]:::plugin_desc
        DCI2["Web-App Template"]:::webapp
        DCI1 --> DCI2
    end
    DSS_Core --> DCI1

    %% Additional Plugins
    subgraph "Additional Plugins"
        %% Dataset Audit Plugin
        subgraph "Dataset Audit Plugin"
            DA1["Code Environment"]:::code_env
            DA2["Custom Recipe"]:::custom_recipe
        end
        %% Compare-Datasets Plugin
        subgraph "Compare-Datasets Plugin"
            CD1["Custom Recipe"]:::custom_recipe
        end
        %% Clubhouse Plugin
        subgraph "Clubhouse Plugin"
            CH1["Connector"]:::connector
        end
        %% Forecastio Plugin
        subgraph "Forecastio Plugin"
            F1["Connector"]:::connector
        end
        %% Disk-Cleanup-Macros Plugin
        subgraph "Disk-Cleanup-Macros Plugin"
            DM1["Runnable"]:::runnable
        end
        %% Microsoft-Power-BI Plugin
        subgraph "Microsoft-Power-BI Plugin"
            MPB1["Web-App Template"]:::webapp
        end
    end
    DSS_Core --> DA1
    DSS_Core --> CD1
    DSS_Core --> CH1
    DSS_Core --> F1
    DSS_Core --> DM1
    DSS_Core --> MPB1

    %% Click Events
    click A1 "https://github.com/dataiku/dataiku-contrib/blob/master/airtable/plugin.json"
    click A2 "https://github.com/dataiku/dataiku-contrib/blob/master/airtable/code-env/python/desc.json"
    click A4 "https://github.com/dataiku/dataiku-contrib/blob/master/airtable/python-connectors/airtable-table/connector.json"
    click A5 "https://github.com/dataiku/dataiku-contrib/blob/master/airtable/python-lib/airtable.py"
    
    click AN1 "https://github.com/dataiku/dataiku-contrib/blob/master/anonymizer/plugin.json"
    
    click CID1 "https://github.com/dataiku/dataiku-contrib/blob/master/clear-intermediate-datasets/plugin.json"
    click CID3 "https://github.com/dataiku/dataiku-contrib/blob/master/clear-intermediate-datasets/python-lib/utils.py"
    click CID4 "https://github.com/dataiku/dataiku-contrib/blob/master/clear-intermediate-datasets/python-runnables/clear_int_datasets/runnable.json"
    
    click DIC1 "https://github.com/dataiku/dataiku-contrib/blob/master/deeplearning-image-cpu/plugin.json"
    click DIC2 "https://github.com/dataiku/dataiku-contrib/blob/master/deeplearning-image-cpu/code-env/python/desc.json"
    click DIC3 "https://github.com/dataiku/dataiku-contrib/blob/master/deeplearning-image-cpu/custom-recipes/deeplearning-image-extract/recipe.json"
    click DIC4 "https://github.com/dataiku/dataiku-contrib/blob/master/deeplearning-image-cpu/python-lib/api_designer_utils.py"
    click DIC5 "https://github.com/dataiku/dataiku-contrib/blob/master/deeplearning-image-cpu/python-lib/constants.py"
    click DIC6 "https://github.com/dataiku/dataiku-contrib/blob/master/deeplearning-image-cpu/python-runnables/create-api-service/runnable.json"
    click DIC7 "https://github.com/dataiku/dataiku-contrib/tree/master/deeplearning-image-cpu/web-app-templates/standard/tensorboard-web-app"
    
    click DCI1 "https://github.com/dataiku/dataiku-contrib/blob/master/dowhy-causal-inference/plugin.json"
    click DCI2 "https://github.com/dataiku/dataiku-contrib/tree/master/dowhy-causal-inference/web-app-templates/standard/causal_inference"
    
    click DA1 "https://github.com/dataiku/dataiku-contrib/blob/master/dataset-audit/code-env/python/desc.json"
    click DA2 "https://github.com/dataiku/dataiku-contrib/blob/master/dataset-audit/custom-recipes/dataset-audit-sqlike/recipe.json"
    click CD1 "https://github.com/dataiku/dataiku-contrib/blob/master/compare-datasets/custom-recipes/compute-difference/recipe.json"
    click CH1 "https://github.com/dataiku/dataiku-contrib/blob/master/clubhouse/python-connectors/clubhouse_epics/connector.json"
    click F1 "https://github.com/dataiku/dataiku-contrib/blob/master/forecastio/python-connectors/forecastio-history/connector.json"
    click DM1 "https://github.com/dataiku/dataiku-contrib/blob/master/disk-cleanup-macros/python-runnables/*/runnable.json"
    click MPB1 "https://github.com/dataiku/dataiku-contrib/tree/master/microsoft-power-bi/web-app-templates/standard/power-bi-management"

    %% Styles
    class DSS_Core dss
    class A1,AN1,CID1,DIC1,DCI1 plugin_desc
    class A2,DIC2,DA1 code_env
    class DIC3,DA2,CD1 custom_recipe
    class A4,CH1,F1 connector
    class A5,CID3,DIC4,DIC5 lib
    class CID4,DIC6,DM1 runnable
    class DIC7,DCI2,MPB1 webapp
```
