How dataiku/dss-plugin-dataiku-pulse works
Dataiku Pulse is a read-only, cross-instance DSS observability product. Workers collect instance/project metadata and audit activity, normalize and store intermediate data, and the hub compacts audit data and builds DuckDB GOLD tables that the dashboard queries for product, asset, user, and activity views. The README documents the overall stages; sampled source and the SOURCE INDEX support selected internal links, while several dashboard and GOLD capabilities are documented but not fully sampled.
Main entry points: recipe.py GOLD recipe · silver.py Silver normalization · app.py Dashboard API · compact_silver_coordinator.py Silver coordinator
Components 24
Worker collection
- Instance metadata
python-runnables/data-gather-instance/runnable.py - Project metadata
python-runnables/data-gather-project/runnable.py - Audit signals
python-runnables/data-gather-audit-logs/runnable.py - Event history
python-runnables/load-event-server-history/runnable.py
Normalize and compact
- Silver normalization
python-lib/data_collection/data_normalizer/silver.py - Type casting
python-lib/data_collection/data_normalizer/casting.py - Silver coordinator
python-lib/data_collection/audit_logs_modules/compact_silver_coordinator.py - Compaction queue
python-lib/data_collection/audit_logs_modules/compact_silver_queue.py - Event replay
python-lib/data_collection/audit_logs_modules/event_mapping_replay.py - Method rules
python-lib/data_collection/method_rules.py
Storage and DuckDB
- Folder discovery
python-lib/shared_storage_discovery.py - Storage context
python-lib/shared_duckdb/context.py - DuckDB setup
python-lib/shared_duckdb/create_conn.py - Parquet engine
python-lib/data_collection/helper/parquet_engine.py
GOLD analytics
- GOLD recipe
custom-recipes/create-gold-tables/recipe.py - GOLD context
python-lib/data_collection/pulse_duckdb/context.py - Object activity
python-lib/data_collection/pulse_duckdb/object_activity.py - GOLD unload
python-lib/data_collection/pulse_duckdb/unload.py
Dashboard
- Dashboard API
webapps/entry_point/app.py - Dashboard connection
webapps/entry_point/pulse_duckdb/engine/create_conn.py - Dashboard queries
webapps/entry_point/pulse_duckdb/engine/query.py - Dashboard interface
webapps/entry_point/frontend/src/App.js - User reports
python-lib/pulse_dashboard/reporting/users_report.py
Other components
- Platform teams
Connections 23
- Platform teams → Dashboard API: explore analytics
- Instance metadata → Silver normalization: normalize outputs
- Project metadata → Silver normalization: normalize outputs
- Audit signals → Silver normalization: normalize signals
- Event history → Silver normalization: normalize events
- Silver normalization → Type casting: casts values
- Audit signals → Silver coordinator: supports compaction
- Silver coordinator → Compaction queue: uses queue
python-lib/data_collection/audit_logs_modules/compact_silver_coordinator.py - Silver coordinator → Event replay: invokes replay
python-lib/data_collection/audit_logs_modules/compact_silver_coordinator.py - Instance metadata → Method rules: applies rules
python-runnables/data-gather-instance/runnable.py - Silver normalization → Folder discovery: selects stored inputs
- GOLD recipe → GOLD context: uses context
custom-recipes/create-gold-tables/recipe.py - GOLD context → Storage context: builds context
python-lib/data_collection/pulse_duckdb/context.py - GOLD recipe → Object activity: builds analytics
custom-recipes/create-gold-tables/recipe.py - GOLD recipe → GOLD unload: unloads tables
custom-recipes/create-gold-tables/recipe.py - GOLD recipe → DuckDB setup: prepares DuckDB
custom-recipes/create-gold-tables/recipe.py - Dashboard API → Dashboard queries: runs queries
webapps/entry_point/app.py - Dashboard queries → Dashboard connection: opens connection
webapps/entry_point/pulse_duckdb/engine/query.py - Dashboard API → Dashboard interface: serves dashboard data
- Dashboard API → GOLD recipe: reads GOLD outputs
- Dashboard API → User reports: offers user reporting
- GOLD recipe → Folder discovery: reads managed storage
- GOLD recipe → Parquet engine: enables Parquet