# youbey/mlsecops-research architecture

> Architecture diagram of the GitHub repository youbey/mlsecops-research, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/youbey/mlsecops-research
- Repository: https://github.com/youbey/mlsecops-research

## Overview

##### Project type and purpose
- This is a research‐grade ML security operations platform (“MLSecOps”) for secured federated learning deployments. It comprises:
- A federated learning (FL) framework (server + clients, honest and malicious)
- Security‐audit modules to detect poisoning, backdoor, inference attacks
- End-to-end infrastructure as code (Ansible + Docker Compose)
- Monitoring/observability stack (Prometheus, Grafana, Loki)
- CI/QA pipeline (Jenkins, bandit, semgrep, pylint, pytest)

##### File structure
Top level
- ansible/ → hosts inventory + playbook to provision hosts & deploy Docker
- fl-project/ → core application, infra definitions, QA rules, scripts, tests
- Jenkinsfile → CI pipeline specification

fl-project/app/
- server.py → FL central aggregator
- client.py & malicious_client.py → honest/malicious FL workers
- audit/*.py → modules to detect poisoning, privacy, Sybil attacks
- structured_logger.py → standardized logging for monitoring

fl-project/infra/
- docker/ → Dockerfile + two docker-compose files (app vs infra)
- monitoring/ → Grafana provisioning, Prometheus & Loki configs

fl-project/qa/ & tests/
- QA rules, security scans, unit/integration tests

##### Review README insights
- Emphasizes MLSecOps concept (extend DevSecOps to ML)
- Provides Ansible setup instructions for provisioning + deployment

1. Federated Learning Layer
– FL Server (aggregator)
– FL Clients (honest + malicious clients running client.py / malicious_client.py)
– Audit Modules (poisoning, backdoor, inference detectors)
2. Data Layer
– Local client datasets (JSON files under fl-project/data)
3. Orchestration & Deployment
– Ansible Control Node → provisions target hosts
– Docker Host(s) running Docker Compose
4. Observability Stack
– Prometheus (metrics)
– Loki (logs)
– Grafana (dashboards)
5. CI/QA Pipeline
– Jenkins (executes Jenkinsfile)
– QA tools (bandit, semgrep, pylint)
– pytest tests

– Clients pull global model ←→ send model updates → Server
– Audit modules subscribe to server/client events → emit security findings
– structured_logger feeds logs → Loki
– application metrics → Prometheus
– Prometheus & Loki → Grafana dashboards
– Ansible → orchestrates & deploys Docker containers for app + infra
– Jenkins → triggers Ansible, runs QA scans, runs tests, publishes reports

c. Architectural patterns & principles
– Agent-server for FL coordination
– Microservice pattern: separate containers for FL server, client, monitoring, audit
– Infrastructure as Code (Ansible + Docker Compose)
– Observability (logs + metrics + dashboards)
– DevSecOps pipeline integrating static analysis and tests

d. Key technologies
– Python (app, audit, scripts)
– Docker + Docker Compose
– Ansible
– Prometheus, Grafana, Loki
– Jenkins, bandit, semgrep, pylint, pytest

##### Diagramming conventions
– Blue for core ML components (server/clients/audit)
– Green for infrastructure (Ansible, Docker)
– Orange for observability (Prometheus, Loki, Grafana)
– Purple for CI/QA pipeline (Jenkins, scanners, tests)

##### Detail
– Capture every container or service defined under fl-project/infra/docker & monitoring
– Include inventory/hosts under ansible to show multi-host provisioning (if applicable)
– Show fetch/split/control scripts as utilities invoked by CI or operators

## Diagram (Mermaid)

```mermaid
flowchart TB
  %% Federated Learning Layer
  subgraph "Federated Learning Layer"
    direction TB
    FLServer["FL Server (Python)"]:::ml
    FLClientHonest["FL Client (Honest, Python)"]:::ml
    FLClientMalicious["FL Client (Malicious, Python)"]:::ml
    subgraph "Security Audit Modules"
      direction TB
      PoisoningDetector["Poisoning Detector"]:::ml
      PrivacyDetector["Privacy Attack Detector"]:::ml
      SybilDetector["Sybil Attack Detector"]:::ml
    end
  end

  %% Data Layer
  subgraph "Data Layer"
    direction TB
    HonestData["Honest Client Data"]:::data
    MaliciousData["Malicious Client Data"]:::data
  end

  %% Orchestration & Deployment
  subgraph "Orchestration & Deployment"
    direction TB
    AnsibleControl["Ansible Control Node"]:::infra
    HostsInventory["hosts.ini"]:::infraform
    AnsibleCFG["ansible.cfg"]:::infraform
    Playbook["site.yml"]:::infraform
    DockerHost["Docker Host (Docker Compose)"]:::infra
  end

  %% Observability Stack
  subgraph "Observability Stack"
    direction TB
    Prometheus["Prometheus (Metrics)"]:::observ
    Loki["Loki (Logs)"]:::observ
    Grafana["Grafana (Dashboards)"]:::observ
  end

  %% CI/QA Pipeline
  subgraph "CI/QA Pipeline"
    direction TB
    Jenkins["Jenkins"]:::ci
    QAtools["QA Tools\n(bandit, semgrep, pylint)"]:::ci
    Pytest["pytest Tests"]:::ci
    ExportMetrics["export_metrics.py"]:::ci
    ControlScript["control.py"]:::ci
    FetchSplit["fetch_and_split.py"]:::ci
    TestsSecurity["test_security_attacks.py"]:::ci
    TestsSAST["vuln_sast_test.py"]:::ci
  end

  %% Relationships
  FLClientHonest -->|pull global model| FLServer
  FLClientMalicious -->|pull global model| FLServer
  FLClientHonest -->|send updates| FLServer
  FLClientMalicious -->|send updates| FLServer
  FLServer -->|broadcast global model| FLClientHonest
  FLServer -->|broadcast global model| FLClientMalicious

  FLClientHonest -->|emit events| PoisoningDetector
  FLClientHonest -->|emit events| PrivacyDetector
  FLClientHonest -->|emit events| SybilDetector
  FLClientMalicious -->|emit events| PoisoningDetector
  FLClientMalicious -->|emit events| PrivacyDetector
  FLClientMalicious -->|emit events| SybilDetector
  FLServer -->|emit events| PoisoningDetector
  FLServer -->|emit events| PrivacyDetector
  FLServer -->|emit events| SybilDetector

  FLClientHonest -->|loads data| HonestData
  FLClientMalicious -->|loads data| MaliciousData

  AnsibleControl -->|uses| HostsInventory
  AnsibleControl -->|uses| AnsibleCFG
  AnsibleControl -->|executes| Playbook
  AnsibleControl -->|provisions| DockerHost

  DockerHost -->|runs| FLServer
  DockerHost -->|runs| FLClientHonest
  DockerHost -->|runs| FLClientMalicious
  DockerHost -->|runs| PoisoningDetector
  DockerHost -->|runs| PrivacyDetector
  DockerHost -->|runs| SybilDetector
  DockerHost -->|runs| structured_logger

  FLServer -->|metrics| Prometheus
  FLClientHonest -->|metrics| Prometheus
  FLClientMalicious -->|metrics| Prometheus

  structured_logger -->|logs| Loki
  Prometheus -->|data| Grafana
  Loki -->|data| Grafana

  Jenkins -->|triggers| AnsibleControl
  Jenkins -->|runs| QAtools
  Jenkins -->|runs| Pytest
  Jenkins -->|executes| ExportMetrics
  Jenkins -->|executes| ControlScript
  Jenkins -->|executes| FetchSplit
  Jenkins -->|runs| TestsSecurity
  Jenkins -->|runs| TestsSAST

  %% Click Events
  click FLServer "https://github.com/youbey/mlsecops-research/blob/master/fl-project/app/server.py"
  click FLClientHonest "https://github.com/youbey/mlsecops-research/blob/master/fl-project/app/client.py"
  click FLClientMalicious "https://github.com/youbey/mlsecops-research/blob/master/fl-project/app/malicious_client.py"
  click PoisoningDetector "https://github.com/youbey/mlsecops-research/blob/master/fl-project/app/audit/detect_poisoning.py"
  click PrivacyDetector "https://github.com/youbey/mlsecops-research/blob/master/fl-project/app/audit/detect_privacy_attacks.py"
  click SybilDetector "https://github.com/youbey/mlsecops-research/blob/master/fl-project/app/audit/detect_sybil_attacks.py"
  click structured_logger "https://github.com/youbey/mlsecops-research/blob/master/fl-project/app/structured_logger.py"
  click HonestData "https://github.com/youbey/mlsecops-research/blob/master/fl-project/data/client_1_data.json"
  click MaliciousData "https://github.com/youbey/mlsecops-research/blob/master/fl-project/data/malicious_client_data.json"
  click AnsibleCFG "https://github.com/youbey/mlsecops-research/blob/master/ansible/ansible.cfg"
  click HostsInventory "https://github.com/youbey/mlsecops-research/blob/master/ansible/inventory/hosts.ini"
  click Playbook "https://github.com/youbey/mlsecops-research/blob/master/ansible/playbooks/site.yml"
  click dockercomposeApp "https://github.com/youbey/mlsecops-research/blob/master/fl-project/infra/docker/docker-compose-app.yml"
  click dockercomposeInfra "https://github.com/youbey/mlsecops-research/blob/master/fl-project/infra/docker/docker-compose-infra.yml"
  click Dockerfile "https://github.com/youbey/mlsecops-research/tree/master/fl-project/infra/docker/Dockerfile"
  click Prometheus "https://github.com/youbey/mlsecops-research/blob/master/fl-project/infra/monitoring/prometheus.yml"
  click Loki "https://github.com/youbey/mlsecops-research/blob/master/fl-project/infra/monitoring/loki-config.yml"
  click Grafana "https://github.com/youbey/mlsecops-research/tree/master/fl-project/infra/monitoring/grafana/"
  click Jenkins "https://github.com/youbey/mlsecops-research/tree/master/Jenkinsfile"
  click QAtools "https://github.com/youbey/mlsecops-research/blob/master/fl-project/qa/.pylintrc"
  click QAtools "https://github.com/youbey/mlsecops-research/blob/master/fl-project/qa/bandit.yml"
  click QAtools "https://github.com/youbey/mlsecops-research/blob/master/fl-project/qa/semgrep-rules.yaml"
  click ExportMetrics "https://github.com/youbey/mlsecops-research/blob/master/fl-project/qa/export_metrics.py"
  click TestsSecurity "https://github.com/youbey/mlsecops-research/blob/master/fl-project/tests/test_security_attacks.py"
  click TestsSAST "https://github.com/youbey/mlsecops-research/blob/master/fl-project/tests/vuln_sast_test.py"
  click ControlScript "https://github.com/youbey/mlsecops-research/blob/master/fl-project/scripts/control.py"
  click FetchSplit "https://github.com/youbey/mlsecops-research/blob/master/fl-project/scripts/fetch_and_split.py"

  %% Styles
  classDef ml fill:#ADD8E6,stroke:#000,stroke-width:1px
  classDef infra fill:#90EE90,stroke:#000,stroke-width:1px
  classDef infraform fill:#98FB98,stroke:#000,stroke-width:1px,stroke-dasharray: 5 5
  classDef data fill:#E0FFFF,stroke:#000,stroke-width:1px
  classDef observ fill:#FFA500,stroke:#000,stroke-width:1px
  classDef ci fill:#DDA0DD,stroke:#000,stroke-width:1px
```
