# dataiku/dss-plugin-dataiku-pulse architecture

> Architecture diagram of the GitHub repository dataiku/dss-plugin-dataiku-pulse, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dataiku/dss-plugin-dataiku-pulse
- Repository: https://github.com/dataiku/dss-plugin-dataiku-pulse

## Overview

Dataiku Pulse is a read-only, cross-instance DSS observability product. Workers collect instance/project metadata and audit activity, normalize and store intermediate data, and the hub compacts audit data and builds DuckDB GOLD tables that the dashboard queries for product, asset, user, and activity views. The README documents the overall stages; sampled source and the SOURCE INDEX support selected internal links, while several dashboard and GOLD capabilities are documented but not fully sampled.

Main entry points: [`custom-recipes/create-gold-tables/recipe.py`](https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/custom-recipes/create-gold-tables/recipe.py) (GOLD recipe), [`python-lib/data_collection/data_normalizer/silver.py`](https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-lib/data_collection/data_normalizer/silver.py) (Silver normalization), [`webapps/entry_point/app.py`](https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/webapps/entry_point/app.py) (Dashboard API), [`python-lib/data_collection/audit_logs_modules/compact_silver_coordinator.py`](https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-lib/data_collection/audit_logs_modules/compact_silver_coordinator.py) (Silver coordinator)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collect["Worker collection"]
  node_instance_collect["Instance metadata<br/>[runnable.py]"]
  node_project_collect["Project metadata<br/>[runnable.py]"]
  node_audit_collect["Audit signals<br/>[runnable.py]"]
  node_history_load["Event history<br/>[runnable.py]"]
end

subgraph group_normalize["Normalize and compact"]
  node_normalizer["Silver normalization<br/>[silver.py]"]
  node_casting["Type casting<br/>[casting.py]"]
  node_compact_coordinator["Silver coordinator"]
  node_compact_queue["Compaction queue"]
  node_event_replay["Event replay"]
  node_method_rules["Method rules<br/>[method_rules.py]"]
end

subgraph group_storage["Storage and DuckDB"]
  node_storage_discovery["Folder discovery"]
  node_storage_context["Storage context<br/>[context.py]"]
  node_duckdb_conn["DuckDB setup<br/>[create_conn.py]"]
  node_parquet_engine["Parquet engine<br/>[parquet_engine.py]"]
end

subgraph group_gold["GOLD analytics"]
  node_gold_recipe["GOLD recipe<br/>[recipe.py]"]
  node_gold_context["GOLD context<br/>[context.py]"]
  node_object_activity["Object activity<br/>[object_activity.py]"]
  node_gold_unload["GOLD unload<br/>[unload.py]"]
end

subgraph group_dashboard["Dashboard"]
  node_dashboard_app["Dashboard API<br/>[app.py]"]
  node_dashboard_conn["Dashboard connection<br/>[create_conn.py]"]
  node_dashboard_query["Dashboard queries<br/>[query.py]"]
  node_frontend["Dashboard interface<br/>[App.js]"]
  node_reporting["User reports<br/>[users_report.py]"]
end

node_admins(("Platform teams"))

node_admins -->|"explore analytics"| node_dashboard_app
node_instance_collect -->|"normalize outputs"| node_normalizer
node_project_collect -->|"normalize outputs"| node_normalizer
node_audit_collect -->|"normalize signals"| node_normalizer
node_history_load -->|"normalize events"| node_normalizer
node_normalizer -->|"casts values"| node_casting
node_audit_collect -.->|"supports compaction"| node_compact_coordinator
node_compact_coordinator -->|"uses queue"| node_compact_queue
node_compact_coordinator -->|"invokes replay"| node_event_replay
node_instance_collect -->|"applies rules"| node_method_rules
node_normalizer -.->|"selects stored inputs"| node_storage_discovery
node_gold_recipe -->|"uses context"| node_gold_context
node_gold_context -->|"builds context"| node_storage_context
node_gold_recipe -->|"builds analytics"| node_object_activity
node_gold_recipe -->|"unloads tables"| node_gold_unload
node_gold_recipe -->|"prepares DuckDB"| node_duckdb_conn
node_dashboard_app -->|"runs queries"| node_dashboard_query
node_dashboard_query -->|"opens connection"| node_dashboard_conn
node_dashboard_app -.->|"serves dashboard data"| node_frontend
node_dashboard_app -.->|"reads GOLD outputs"| node_gold_recipe
node_dashboard_app -.->|"offers user reporting"| node_reporting
node_gold_recipe -.->|"reads managed storage"| node_storage_discovery
node_gold_recipe -.->|"enables Parquet"| node_parquet_engine

click node_instance_collect "https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-runnables/data-gather-instance/runnable.py"
click node_project_collect "https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-runnables/data-gather-project/runnable.py"
click node_audit_collect "https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-runnables/data-gather-audit-logs/runnable.py"
click node_history_load "https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-runnables/load-event-server-history/runnable.py"
click node_normalizer "https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-lib/data_collection/data_normalizer/silver.py"
click node_casting "https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-lib/data_collection/data_normalizer/casting.py"
click node_compact_coordinator "https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-lib/data_collection/audit_logs_modules/compact_silver_coordinator.py"
click node_compact_queue "https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-lib/data_collection/audit_logs_modules/compact_silver_queue.py"
click node_event_replay "https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-lib/data_collection/audit_logs_modules/event_mapping_replay.py"
click node_method_rules "https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-lib/data_collection/method_rules.py"
click node_storage_discovery "https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-lib/shared_storage_discovery.py"
click node_storage_context "https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-lib/shared_duckdb/context.py"
click node_duckdb_conn "https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-lib/shared_duckdb/create_conn.py"
click node_parquet_engine "https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-lib/data_collection/helper/parquet_engine.py"
click node_gold_recipe "https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/custom-recipes/create-gold-tables/recipe.py"
click node_gold_context "https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-lib/data_collection/pulse_duckdb/context.py"
click node_object_activity "https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-lib/data_collection/pulse_duckdb/object_activity.py"
click node_gold_unload "https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-lib/data_collection/pulse_duckdb/unload.py"
click node_dashboard_app "https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/webapps/entry_point/app.py"
click node_dashboard_conn "https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/webapps/entry_point/pulse_duckdb/engine/create_conn.py"
click node_dashboard_query "https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/webapps/entry_point/pulse_duckdb/engine/query.py"
click node_frontend "https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/webapps/entry_point/frontend/src/App.js"
click node_reporting "https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-lib/pulse_dashboard/reporting/users_report.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_instance_collect,node_project_collect,node_audit_collect,node_history_load toneBlue
class node_normalizer,node_casting,node_compact_coordinator,node_compact_queue,node_event_replay,node_method_rules toneAmber
class node_storage_discovery,node_storage_context,node_duckdb_conn,node_parquet_engine toneMint
class node_gold_recipe,node_gold_context,node_object_activity,node_gold_unload toneRose
class node_dashboard_app,node_dashboard_conn,node_dashboard_query,node_frontend,node_reporting,node_admins toneIndigo
```

## Components

### Worker collection

- [Instance metadata](https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-runnables/data-gather-instance/runnable.py): `python-runnables/data-gather-instance/runnable.py`
- [Project metadata](https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-runnables/data-gather-project/runnable.py): `python-runnables/data-gather-project/runnable.py`
- [Audit signals](https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-runnables/data-gather-audit-logs/runnable.py): `python-runnables/data-gather-audit-logs/runnable.py`
- [Event history](https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-runnables/load-event-server-history/runnable.py): `python-runnables/load-event-server-history/runnable.py`

### Normalize and compact

- [Silver normalization](https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-lib/data_collection/data_normalizer/silver.py): `python-lib/data_collection/data_normalizer/silver.py`
- [Type casting](https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-lib/data_collection/data_normalizer/casting.py): `python-lib/data_collection/data_normalizer/casting.py`
- [Silver coordinator](https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-lib/data_collection/audit_logs_modules/compact_silver_coordinator.py): `python-lib/data_collection/audit_logs_modules/compact_silver_coordinator.py`
- [Compaction queue](https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-lib/data_collection/audit_logs_modules/compact_silver_queue.py): `python-lib/data_collection/audit_logs_modules/compact_silver_queue.py`
- [Event replay](https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-lib/data_collection/audit_logs_modules/event_mapping_replay.py): `python-lib/data_collection/audit_logs_modules/event_mapping_replay.py`
- [Method rules](https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-lib/data_collection/method_rules.py): `python-lib/data_collection/method_rules.py`

### Storage and DuckDB

- [Folder discovery](https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-lib/shared_storage_discovery.py): `python-lib/shared_storage_discovery.py`
- [Storage context](https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-lib/shared_duckdb/context.py): `python-lib/shared_duckdb/context.py`
- [DuckDB setup](https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-lib/shared_duckdb/create_conn.py): `python-lib/shared_duckdb/create_conn.py`
- [Parquet engine](https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-lib/data_collection/helper/parquet_engine.py): `python-lib/data_collection/helper/parquet_engine.py`

### GOLD analytics

- [GOLD recipe](https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/custom-recipes/create-gold-tables/recipe.py): `custom-recipes/create-gold-tables/recipe.py`
- [GOLD context](https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-lib/data_collection/pulse_duckdb/context.py): `python-lib/data_collection/pulse_duckdb/context.py`
- [Object activity](https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-lib/data_collection/pulse_duckdb/object_activity.py): `python-lib/data_collection/pulse_duckdb/object_activity.py`
- [GOLD unload](https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-lib/data_collection/pulse_duckdb/unload.py): `python-lib/data_collection/pulse_duckdb/unload.py`

### Dashboard

- [Dashboard API](https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/webapps/entry_point/app.py): `webapps/entry_point/app.py`
- [Dashboard connection](https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/webapps/entry_point/pulse_duckdb/engine/create_conn.py): `webapps/entry_point/pulse_duckdb/engine/create_conn.py`
- [Dashboard queries](https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/webapps/entry_point/pulse_duckdb/engine/query.py): `webapps/entry_point/pulse_duckdb/engine/query.py`
- [Dashboard interface](https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/webapps/entry_point/frontend/src/App.js): `webapps/entry_point/frontend/src/App.js`
- [User reports](https://github.com/dataiku/dss-plugin-dataiku-pulse/blob/main/python-lib/pulse_dashboard/reporting/users_report.py): `python-lib/pulse_dashboard/reporting/users_report.py`

### Other components

- **Platform teams**

## Connections

- Platform teams → Dashboard API: explore analytics
- Instance metadata → Silver normalization: normalize outputs
- Project metadata → Silver normalization: normalize outputs
- Audit signals → Silver normalization: normalize signals
- Event history → Silver normalization: normalize events
- Silver normalization → Type casting: casts values
- Audit signals → Silver coordinator: supports compaction
- Silver coordinator → Compaction queue: uses queue (evidence: `python-lib/data_collection/audit_logs_modules/compact_silver_coordinator.py`)
- Silver coordinator → Event replay: invokes replay (evidence: `python-lib/data_collection/audit_logs_modules/compact_silver_coordinator.py`)
- Instance metadata → Method rules: applies rules (evidence: `python-runnables/data-gather-instance/runnable.py`)
- Silver normalization → Folder discovery: selects stored inputs
- GOLD recipe → GOLD context: uses context (evidence: `custom-recipes/create-gold-tables/recipe.py`)
- GOLD context → Storage context: builds context (evidence: `python-lib/data_collection/pulse_duckdb/context.py`)
- GOLD recipe → Object activity: builds analytics (evidence: `custom-recipes/create-gold-tables/recipe.py`)
- GOLD recipe → GOLD unload: unloads tables (evidence: `custom-recipes/create-gold-tables/recipe.py`)
- GOLD recipe → DuckDB setup: prepares DuckDB (evidence: `custom-recipes/create-gold-tables/recipe.py`)
- Dashboard API → Dashboard queries: runs queries (evidence: `webapps/entry_point/app.py`)
- Dashboard queries → Dashboard connection: opens connection (evidence: `webapps/entry_point/pulse_duckdb/engine/query.py`)
- Dashboard API → Dashboard interface: serves dashboard data
- Dashboard API → GOLD recipe: reads GOLD outputs
- Dashboard API → User reports: offers user reporting
- GOLD recipe → Folder discovery: reads managed storage
- GOLD recipe → Parquet engine: enables Parquet
