How neha-jacob-8/insider-threat-detection-bda works
This repository implements a CERT insider-threat detection pipeline plus two dashboard clients. The documented batch workflow moves activity data through cleaning, user organization, feature/graph analysis, LSTM scoring and risk fusion into APIs and dashboard views; separate Kafka paths support live ingestion. The excerpts directly show Spark streaming writes to HDFS, daily feature construction, UI page composition and sample API routes. Several batch scoring and graph stages are present in the tree and README but their source was not sampled, so their internal links are not asserted.
Main entry points: BuildUserDailyFeatures.scala Daily features · ScoreFusion.scala Risk fusion · BuildMasterDashboardData.scala Dashboard datasets · kaggle_to_kafka.py Kaggle ingestion
Components 24
Data ingestion
- CERT activity data
- Kaggle ingestion
kaggle_to_kafka.py - Kafka topics
- CSV Kafka producer
kafka_producer.py - Spark stream ingest
spark_streaming_ingest.py
Data preparation
- HDFS data lake
- Event cleaning
preprocessing/src/main/scala/clean - User partitioning
preprocessing/src/main/scala/organize
Threat analytics
- Daily features
preprocessing/src/main/scala/features/BuildUserDailyFeatures.scala - Graph analytics
preprocessing/src/main/scala/features - LSTM training
models/lstm/train_lstm_autoencoder.py - LSTM scores
models/lstm/evaluate_lstm_scores.py - Risk fusion
preprocessing/src/main/scala/features/ScoreFusion.scala - Risk thresholds
preprocessing/src/main/scala/features/RiskThresholding.scala
Serving layer
- Dashboard datasets
preprocessing/src/main/scala/features/BuildMasterDashboardData.scala - FastAPI service
api.py - JSON export
export_to_json.py - Timeline extraction
extract_timeline.py - Dashboard mock API
insider-threat-detection-dashboard/server.ts
Dashboards
- React application
insider-threat-detection-dashboard/src/App.tsx - Dashboard views
insider-threat-detection-dashboard/src/components/pages - Dashboard API client
insider-threat-detection-dashboard/src/services/api.ts - Vanilla dashboard
dashboard/app.js
Other components
- Analyst
Connections 31
- CERT activity data → Kaggle ingestion: ingests
README.md - CERT activity data → CSV Kafka producer: streams CSV
- Kaggle ingestion → Kafka topics: publishes events
- CSV Kafka producer → Kafka topics: publishes events
- Kafka topics → Spark stream ingest: feeds stream
spark_streaming_ingest.py - Spark stream ingest → HDFS data lake: writes Parquet
spark_streaming_ingest.py - CERT activity data → HDFS data lake: uploads raw data
README.md - HDFS data lake → Event cleaning: reads raw logs
README.md - Event cleaning → HDFS data lake: writes clean Parquet
README.md - HDFS data lake → User partitioning: reads cleaned data
README.md - User partitioning → HDFS data lake: writes user partitions
README.md - HDFS data lake → Daily features: reads event tables
preprocessing/src/main/scala/features/BuildUserDailyFeatures.scala - Daily features → HDFS data lake: writes features
preprocessing/src/main/scala/features/BuildUserDailyFeatures.scala - HDFS data lake → LSTM training: supplies sequences
models/lstm/train_lstm_autoencoder.py - LSTM training → HDFS data lake: stores model
models/lstm/train_lstm_autoencoder.py - Daily features → Graph analytics: supports graph scoring
- Daily features → Risk fusion: contributes scores
- Graph analytics → Risk fusion: contributes scores
- LSTM scores → Risk fusion: contributes scores
- Risk fusion → Risk thresholds: fuses risk
- Risk thresholds → Dashboard datasets: provides risk results
- HDFS data lake → Dashboard datasets: supplies analytics data
- Dashboard datasets → FastAPI service: serves results
- HDFS data lake → JSON export: exports data
- HDFS data lake → Timeline extraction: extracts activity
- FastAPI service → Vanilla dashboard: answers API requests
dashboard/app.js - React application → Dashboard views: renders views
insider-threat-detection-dashboard/src/App.tsx - Dashboard views → Dashboard API client: requests dashboard data
- Dashboard API client → Dashboard mock API: calls endpoints
- Analyst → React application: uses dashboard
- Analyst → Vanilla dashboard: uses dashboard