How dataiku/dataiku-contrib works
Main Components
• Plugins: Each sub-directory (e.g., “airtable”, “anonymizer”, “clear-intermediate-datasets”, “deeplearning-image-cpu”, “dowhy-causal-inference”, “forecastio”, etc.) represents an individual plugin. Although there are many plugins, they’re all based on a common plugin architecture pattern.
• Plugin Components: For each plugin module, identify common sub-components such as:
– Plugin Descriptor (plugin.json) – which defines the plugin’s manifest and metadata.
– Custom Recipes – locations where new functionalities (e.g., scoring, prediction, enrichment, etc.) are implemented (typically under directories like custom-recipes).
– Connectors – components under python-connectors (or their language equivalent) that enable integration with external systems.
– Code Environments and Libraries – directories like code-env, python-lib, and similar folders that define runtime environments and include common/shared code.
– Runnables – directories that provide executable scripts (python-runnables, etc.) supporting custom operations.
Relationships and Workflows
• Within each plugin, illustrate the internal separation: how the descriptor defines the interface, while custom recipes and connectors form the implementation modules.
• Integration pattern: DSS (the central engine) acts as the orchestrator, loading, validating, and running the plugins during different stages of a data science workflow.
Architectural Patterns and Design Principles
• Plugin-based Design: Clearly highlight the extensibility principle—each plugin is independently developed yet adheres to a common structure that DSS can discover and integrate.
• Separation of Concerns: The design should show the decoupling between presentation elements (if any JavaScript or web-app templates are involved), execution logic (runnables, connectors), and configuration (plugin.json and code-env descriptors).
• Language and Environment Isolation: Since plugins may use different languages (Python, Java, JavaScript), indicate that the architecture supports heterogeneous technology stacks by having isolated environments per plugin.
Specific Technologies and Libraries
• Fact that many plugins rely on Python (with defined requirements in code-env/python/spec/requirements.txt or similar), as well as Java and JavaScript components.
• Mention that some plugins include web-app templates and resources (CSS, HTML, JS files), indicating front-end integrations.
• Although each plugin might have its own specific libraries, the recurring presence of standardized directories (like python-connectors, python-lib, custom-recipes) indicates adherence to common Dataiku DSS plugin development practices.