# abdulrehman25/msc-finals architecture

> Architecture diagram of the GitHub repository abdulrehman25/msc-finals, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/abdulrehman25/msc-finals
- Repository: https://github.com/abdulrehman25/msc-finals

## Overview

This repository detects anomalies in web access logs with two reconstruction-based branches: per-request content scoring and IP/user-agent session-window scoring. The principal runtime flow is a submitted log line, parsing and feature extraction, branch scores, weighted fusion, and a stored event that the dashboard can display. Separate training commands produce branch models and thresholds; a calibration command sets fusion weights. The README documents these capabilities; evaluation is also exposed by the API. Some API implementation is omitted, so only documented or excerpt-supported wiring is shown.

Main entry points: [`src/api/main.py`](https://github.com/abdulrehman25/msc-finals/blob/main/src/api/main.py) (Scoring API), [`train_session_lstm_vae.py`](https://github.com/abdulrehman25/msc-finals/blob/main/train_session_lstm_vae.py) (Session trainer), [`train_content_ae.py`](https://github.com/abdulrehman25/msc-finals/blob/main/train_content_ae.py) (Content trainer), [`src/featurization/features.py`](https://github.com/abdulrehman25/msc-finals/blob/main/src/featurization/features.py) (Request features)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_input["Log input"]
  node_logfile["Access log"]
  node_parser["Log parser<br/>[parse.py]"]
  node_features["Request features<br/>[features.py]"]
  node_loaders["Log vector loader<br/>[loaders.py]"]
end

subgraph group_training["Branch training"]
  node_content_train["Content trainer"]
  node_content_model["Content AE"]
  node_session_train["Session trainer"]
  node_sessionize["Session windows<br/>[sessionize.py]"]
  node_session_model["Session LSTM-VAE"]
end

subgraph group_detection["Detection runtime"]
  node_fusion_calibration["Fusion calibration<br/>[fuse_calibrate.py]"]
  node_api["Scoring API<br/>[main.py]"]
end

subgraph group_operations["Review and evaluation"]
  node_event_store[("Event database<br/>[db.py]")]
  node_dashboard["Dashboard<br/>[main.py]"]
  node_evaluation["File evaluation<br/>[evaluate_file.py]"]
  node_metrics["Operating metrics<br/>[metrics_extra.py]"]
end

node_operator(("Log operator"))
node_client(("API client"))

node_operator -->|"provides"| node_logfile
node_logfile -->|"trains from"| node_content_train
node_logfile -->|"trains from"| node_session_train
node_content_train -->|"loads vectors"| node_loaders
node_session_train -->|"loads vectors"| node_loaders
node_content_train -->|"trains"| node_content_model
node_session_train -->|"builds windows"| node_sessionize
node_session_train -->|"trains"| node_session_model
node_content_train -->|"provides scores"| node_fusion_calibration
node_session_train -->|"provides scores"| node_fusion_calibration
node_client -->|"submits line"| node_api
node_api -->|"parses line"| node_parser
node_api -->|"extracts vector"| node_features
node_features -->|"parses line"| node_parser
node_api -->|"scores branch"| node_content_model
node_api -->|"scores branch"| node_session_model
node_api -->|"writes event"| node_event_store
node_api -->|"runs evaluation"| node_evaluation
node_evaluation -.->|"computes metrics"| node_metrics
node_client -->|"views results"| node_dashboard
node_dashboard -->|"reads events"| node_event_store
node_dashboard -->|"requests dashboard data"| node_api

click node_parser "https://github.com/abdulrehman25/msc-finals/blob/main/src/featurization/parse.py"
click node_features "https://github.com/abdulrehman25/msc-finals/blob/main/src/featurization/features.py"
click node_loaders "https://github.com/abdulrehman25/msc-finals/blob/main/src/featurization/loaders.py"
click node_content_train "https://github.com/abdulrehman25/msc-finals/blob/main/train_content_ae.py"
click node_content_model "https://github.com/abdulrehman25/msc-finals/blob/main/src/models/content_autoencoder.py"
click node_session_train "https://github.com/abdulrehman25/msc-finals/blob/main/train_session_lstm_vae.py"
click node_sessionize "https://github.com/abdulrehman25/msc-finals/blob/main/src/pipeline/sessionize.py"
click node_session_model "https://github.com/abdulrehman25/msc-finals/blob/main/src/models/session_lstm_vae.py"
click node_fusion_calibration "https://github.com/abdulrehman25/msc-finals/blob/main/src/pipeline/fuse_calibrate.py"
click node_api "https://github.com/abdulrehman25/msc-finals/blob/main/src/api/main.py"
click node_event_store "https://github.com/abdulrehman25/msc-finals/blob/main/src/storage/db.py"
click node_dashboard "https://github.com/abdulrehman25/msc-finals/blob/main/src/api/main.py"
click node_evaluation "https://github.com/abdulrehman25/msc-finals/blob/main/scripts/evaluate_file.py"
click node_metrics "https://github.com/abdulrehman25/msc-finals/blob/main/src/pipeline/metrics_extra.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_logfile,node_parser,node_features,node_loaders,node_client toneBlue
class node_content_train,node_content_model,node_session_train,node_sessionize,node_session_model toneAmber
class node_fusion_calibration,node_api toneMint
class node_event_store,node_dashboard,node_evaluation,node_metrics toneRose
class node_operator toneIndigo
```

## Components

### Log input

- **Access log**
- [Log parser](https://github.com/abdulrehman25/msc-finals/blob/main/src/featurization/parse.py): `src/featurization/parse.py`
- [Request features](https://github.com/abdulrehman25/msc-finals/blob/main/src/featurization/features.py): `src/featurization/features.py`
- [Log vector loader](https://github.com/abdulrehman25/msc-finals/blob/main/src/featurization/loaders.py): `src/featurization/loaders.py`

### Branch training

- [Content trainer](https://github.com/abdulrehman25/msc-finals/blob/main/train_content_ae.py): `train_content_ae.py`
- [Content AE](https://github.com/abdulrehman25/msc-finals/blob/main/src/models/content_autoencoder.py): `src/models/content_autoencoder.py`
- [Session trainer](https://github.com/abdulrehman25/msc-finals/blob/main/train_session_lstm_vae.py): `train_session_lstm_vae.py`
- [Session windows](https://github.com/abdulrehman25/msc-finals/blob/main/src/pipeline/sessionize.py): `src/pipeline/sessionize.py`
- [Session LSTM-VAE](https://github.com/abdulrehman25/msc-finals/blob/main/src/models/session_lstm_vae.py): `src/models/session_lstm_vae.py`

### Detection runtime

- [Fusion calibration](https://github.com/abdulrehman25/msc-finals/blob/main/src/pipeline/fuse_calibrate.py): `src/pipeline/fuse_calibrate.py`
- [Scoring API](https://github.com/abdulrehman25/msc-finals/blob/main/src/api/main.py): `src/api/main.py`

### Review and evaluation

- [Event database](https://github.com/abdulrehman25/msc-finals/blob/main/src/storage/db.py): `src/storage/db.py`
- [Dashboard](https://github.com/abdulrehman25/msc-finals/blob/main/src/api/main.py): `src/api/main.py`
- [File evaluation](https://github.com/abdulrehman25/msc-finals/blob/main/scripts/evaluate_file.py): `scripts/evaluate_file.py`
- [Operating metrics](https://github.com/abdulrehman25/msc-finals/blob/main/src/pipeline/metrics_extra.py): `src/pipeline/metrics_extra.py`

### Other components

- **Log operator**
- **API client**

## Connections

- Log operator → Access log: provides
- Access log → Content trainer: trains from
- Access log → Session trainer: trains from
- Content trainer → Log vector loader: loads vectors
- Session trainer → Log vector loader: loads vectors
- Content trainer → Content AE: trains
- Session trainer → Session windows: builds windows
- Session trainer → Session LSTM-VAE: trains
- Content trainer → Fusion calibration: provides scores
- Session trainer → Fusion calibration: provides scores
- API client → Scoring API: submits line
- Scoring API → Log parser: parses line
- Scoring API → Request features: extracts vector
- Request features → Log parser: parses line
- Scoring API → Content AE: scores branch
- Scoring API → Session LSTM-VAE: scores branch
- Scoring API → Event database: writes event
- Scoring API → File evaluation: runs evaluation
- File evaluation → Operating metrics: computes metrics
- API client → Dashboard: views results
- Dashboard → Event database: reads events
- Dashboard → Scoring API: requests dashboard data