# neha-jacob-8/insider-threat-detection-bda architecture

> Architecture diagram of the GitHub repository neha-jacob-8/insider-threat-detection-bda, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/neha-jacob-8/insider-threat-detection-bda
- Repository: https://github.com/neha-jacob-8/insider-threat-detection-bda

## Overview

This repository implements a CERT insider-threat detection pipeline plus two dashboard clients. The documented batch workflow moves activity data through cleaning, user organization, feature/graph analysis, LSTM scoring and risk fusion into APIs and dashboard views; separate Kafka paths support live ingestion. The excerpts directly show Spark streaming writes to HDFS, daily feature construction, UI page composition and sample API routes. Several batch scoring and graph stages are present in the tree and README but their source was not sampled, so their internal links are not asserted.

Main entry points: [`preprocessing/src/main/scala/features/BuildUserDailyFeatures.scala`](https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/preprocessing/src/main/scala/features/BuildUserDailyFeatures.scala) (Daily features), [`preprocessing/src/main/scala/features/ScoreFusion.scala`](https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/preprocessing/src/main/scala/features/ScoreFusion.scala) (Risk fusion), [`preprocessing/src/main/scala/features/BuildMasterDashboardData.scala`](https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/preprocessing/src/main/scala/features/BuildMasterDashboardData.scala) (Dashboard datasets), [`kaggle_to_kafka.py`](https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/kaggle_to_kafka.py) (Kaggle ingestion)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Data ingestion"]
  node_dataset["CERT activity data"]
  node_kaggle_ingest["Kaggle ingestion<br/>[kaggle_to_kafka.py]"]
  node_kafka["Kafka topics"]
  node_producer["CSV Kafka producer<br/>[kafka_producer.py]"]
  node_stream_ingest["Spark stream ingest"]
end

subgraph group_prep["Data preparation"]
  node_hdfs[("HDFS data lake")]
  node_cleaning["Event cleaning"]
  node_partitioning["User partitioning"]
end

subgraph group_analytics["Threat analytics"]
  node_daily_features["Daily features"]
  node_graph_features["Graph analytics"]
  node_lstm_train["LSTM training"]
  node_lstm_scores["LSTM scores"]
  node_score_fusion["Risk fusion<br/>[ScoreFusion.scala]"]
  node_thresholding["Risk thresholds"]
end

subgraph group_serving["Serving layer"]
  node_dashboard_data["Dashboard datasets"]
  node_api["FastAPI service<br/>[api.py]"]
  node_json_export["JSON export<br/>[export_to_json.py]"]
  node_timeline["Timeline extraction"]
  node_mock_server["Dashboard mock API<br/>[server.ts]"]
end

subgraph group_dashboard["Dashboards"]
  node_react_app["React application<br/>[App.tsx]"]
  node_react_pages["Dashboard views"]
  node_api_client["Dashboard API client<br/>[api.ts]"]
  node_vanilla_ui["Vanilla dashboard<br/>[app.js]"]
end

node_user(("Analyst"))

node_dataset -->|"ingests"| node_kaggle_ingest
node_dataset -.->|"streams CSV"| node_producer
node_kaggle_ingest -.->|"publishes events"| node_kafka
node_producer -.->|"publishes events"| node_kafka
node_kafka -->|"feeds stream"| node_stream_ingest
node_stream_ingest -->|"writes Parquet"| node_hdfs
node_dataset -->|"uploads raw data"| node_hdfs
node_hdfs -->|"reads raw logs"| node_cleaning
node_cleaning -->|"writes clean Parquet"| node_hdfs
node_hdfs -->|"reads cleaned data"| node_partitioning
node_partitioning -->|"writes user partitions"| node_hdfs
node_hdfs -->|"reads event tables"| node_daily_features
node_daily_features -->|"writes features"| node_hdfs
node_hdfs -->|"supplies sequences"| node_lstm_train
node_lstm_train -->|"stores model"| node_hdfs
node_daily_features -.->|"supports graph scoring"| node_graph_features
node_daily_features -.->|"contributes scores"| node_score_fusion
node_graph_features -.->|"contributes scores"| node_score_fusion
node_lstm_scores -.->|"contributes scores"| node_score_fusion
node_score_fusion -.->|"fuses risk"| node_thresholding
node_thresholding -.->|"provides risk results"| node_dashboard_data
node_hdfs -.->|"supplies analytics data"| node_dashboard_data
node_dashboard_data -.->|"serves results"| node_api
node_hdfs -.->|"exports data"| node_json_export
node_hdfs -.->|"extracts activity"| node_timeline
node_api -->|"answers API requests"| node_vanilla_ui
node_react_app -->|"renders views"| node_react_pages
node_react_pages -.->|"requests dashboard data"| node_api_client
node_api_client -.->|"calls endpoints"| node_mock_server
node_user -.->|"uses dashboard"| node_react_app
node_user -.->|"uses dashboard"| node_vanilla_ui

click node_kaggle_ingest "https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/kaggle_to_kafka.py"
click node_producer "https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/kafka_producer.py"
click node_stream_ingest "https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/spark_streaming_ingest.py"
click node_cleaning "https://github.com/neha-jacob-8/insider-threat-detection-bda/tree/main/preprocessing/src/main/scala/clean"
click node_partitioning "https://github.com/neha-jacob-8/insider-threat-detection-bda/tree/main/preprocessing/src/main/scala/organize"
click node_daily_features "https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/preprocessing/src/main/scala/features/BuildUserDailyFeatures.scala"
click node_graph_features "https://github.com/neha-jacob-8/insider-threat-detection-bda/tree/main/preprocessing/src/main/scala/features"
click node_lstm_train "https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/models/lstm/train_lstm_autoencoder.py"
click node_lstm_scores "https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/models/lstm/evaluate_lstm_scores.py"
click node_score_fusion "https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/preprocessing/src/main/scala/features/ScoreFusion.scala"
click node_thresholding "https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/preprocessing/src/main/scala/features/RiskThresholding.scala"
click node_dashboard_data "https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/preprocessing/src/main/scala/features/BuildMasterDashboardData.scala"
click node_api "https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/api.py"
click node_json_export "https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/export_to_json.py"
click node_timeline "https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/extract_timeline.py"
click node_react_app "https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/insider-threat-detection-dashboard/src/App.tsx"
click node_react_pages "https://github.com/neha-jacob-8/insider-threat-detection-bda/tree/main/insider-threat-detection-dashboard/src/components/pages"
click node_api_client "https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/insider-threat-detection-dashboard/src/services/api.ts"
click node_mock_server "https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/insider-threat-detection-dashboard/server.ts"
click node_vanilla_ui "https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/dashboard/app.js"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dataset,node_kaggle_ingest,node_kafka,node_producer,node_stream_ingest toneBlue
class node_hdfs,node_cleaning,node_partitioning toneAmber
class node_daily_features,node_graph_features,node_lstm_train,node_lstm_scores,node_score_fusion,node_thresholding toneMint
class node_dashboard_data,node_api,node_json_export,node_timeline,node_mock_server toneRose
class node_react_app,node_react_pages,node_api_client,node_vanilla_ui,node_user toneIndigo
```

## Components

### Data ingestion

- **CERT activity data**
- [Kaggle ingestion](https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/kaggle_to_kafka.py): `kaggle_to_kafka.py`
- **Kafka topics**
- [CSV Kafka producer](https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/kafka_producer.py): `kafka_producer.py`
- [Spark stream ingest](https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/spark_streaming_ingest.py): `spark_streaming_ingest.py`

### Data preparation

- **HDFS data lake**
- [Event cleaning](https://github.com/neha-jacob-8/insider-threat-detection-bda/tree/main/preprocessing/src/main/scala/clean): `preprocessing/src/main/scala/clean`
- [User partitioning](https://github.com/neha-jacob-8/insider-threat-detection-bda/tree/main/preprocessing/src/main/scala/organize): `preprocessing/src/main/scala/organize`

### Threat analytics

- [Daily features](https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/preprocessing/src/main/scala/features/BuildUserDailyFeatures.scala): `preprocessing/src/main/scala/features/BuildUserDailyFeatures.scala`
- [Graph analytics](https://github.com/neha-jacob-8/insider-threat-detection-bda/tree/main/preprocessing/src/main/scala/features): `preprocessing/src/main/scala/features`
- [LSTM training](https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/models/lstm/train_lstm_autoencoder.py): `models/lstm/train_lstm_autoencoder.py`
- [LSTM scores](https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/models/lstm/evaluate_lstm_scores.py): `models/lstm/evaluate_lstm_scores.py`
- [Risk fusion](https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/preprocessing/src/main/scala/features/ScoreFusion.scala): `preprocessing/src/main/scala/features/ScoreFusion.scala`
- [Risk thresholds](https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/preprocessing/src/main/scala/features/RiskThresholding.scala): `preprocessing/src/main/scala/features/RiskThresholding.scala`

### Serving layer

- [Dashboard datasets](https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/preprocessing/src/main/scala/features/BuildMasterDashboardData.scala): `preprocessing/src/main/scala/features/BuildMasterDashboardData.scala`
- [FastAPI service](https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/api.py): `api.py`
- [JSON export](https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/export_to_json.py): `export_to_json.py`
- [Timeline extraction](https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/extract_timeline.py): `extract_timeline.py`
- [Dashboard mock API](https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/insider-threat-detection-dashboard/server.ts): `insider-threat-detection-dashboard/server.ts`

### Dashboards

- [React application](https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/insider-threat-detection-dashboard/src/App.tsx): `insider-threat-detection-dashboard/src/App.tsx`
- [Dashboard views](https://github.com/neha-jacob-8/insider-threat-detection-bda/tree/main/insider-threat-detection-dashboard/src/components/pages): `insider-threat-detection-dashboard/src/components/pages`
- [Dashboard API client](https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/insider-threat-detection-dashboard/src/services/api.ts): `insider-threat-detection-dashboard/src/services/api.ts`
- [Vanilla dashboard](https://github.com/neha-jacob-8/insider-threat-detection-bda/blob/main/dashboard/app.js): `dashboard/app.js`

### Other components

- **Analyst**

## Connections

- CERT activity data → Kaggle ingestion: ingests (evidence: `README.md`)
- CERT activity data → CSV Kafka producer: streams CSV
- Kaggle ingestion → Kafka topics: publishes events
- CSV Kafka producer → Kafka topics: publishes events
- Kafka topics → Spark stream ingest: feeds stream (evidence: `spark_streaming_ingest.py`)
- Spark stream ingest → HDFS data lake: writes Parquet (evidence: `spark_streaming_ingest.py`)
- CERT activity data → HDFS data lake: uploads raw data (evidence: `README.md`)
- HDFS data lake → Event cleaning: reads raw logs (evidence: `README.md`)
- Event cleaning → HDFS data lake: writes clean Parquet (evidence: `README.md`)
- HDFS data lake → User partitioning: reads cleaned data (evidence: `README.md`)
- User partitioning → HDFS data lake: writes user partitions (evidence: `README.md`)
- HDFS data lake → Daily features: reads event tables (evidence: `preprocessing/src/main/scala/features/BuildUserDailyFeatures.scala`)
- Daily features → HDFS data lake: writes features (evidence: `preprocessing/src/main/scala/features/BuildUserDailyFeatures.scala`)
- HDFS data lake → LSTM training: supplies sequences (evidence: `models/lstm/train_lstm_autoencoder.py`)
- LSTM training → HDFS data lake: stores model (evidence: `models/lstm/train_lstm_autoencoder.py`)
- Daily features → Graph analytics: supports graph scoring
- Daily features → Risk fusion: contributes scores
- Graph analytics → Risk fusion: contributes scores
- LSTM scores → Risk fusion: contributes scores
- Risk fusion → Risk thresholds: fuses risk
- Risk thresholds → Dashboard datasets: provides risk results
- HDFS data lake → Dashboard datasets: supplies analytics data
- Dashboard datasets → FastAPI service: serves results
- HDFS data lake → JSON export: exports data
- HDFS data lake → Timeline extraction: extracts activity
- FastAPI service → Vanilla dashboard: answers API requests (evidence: `dashboard/app.js`)
- React application → Dashboard views: renders views (evidence: `insider-threat-detection-dashboard/src/App.tsx`)
- Dashboard views → Dashboard API client: requests dashboard data
- Dashboard API client → Dashboard mock API: calls endpoints
- Analyst → React application: uses dashboard
- Analyst → Vanilla dashboard: uses dashboard
