# mm-cyberlabs/m2-fake-sphere architecture

> Architecture diagram of the GitHub repository mm-cyberlabs/m2-fake-sphere, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mm-cyberlabs/m2-fake-sphere
- Repository: https://github.com/mm-cyberlabs/m2-fake-sphere

## Overview

##### Project type and purpose
- This is a Python-based, CLI-first synthetic data engine (“Fake Sphere”) whose goal is to generate realistic test data across databases, APIs, streams, files, and agent flows without using real data.
- It’s packaged as a library/CLI tool (`m2_fake_sphere`), includes scraper/Markdown converter, multiple data-generation backends (Faker, SDV, scikit-learn), and an LLM tie-breaker fine-tune pipeline (Phi-2 + QLoRA/Axolotl).

##### File structure
- Top-level:
- `m2_fake_sphere/` is the Python package: `pyproject.toml`, `src/m2_fake_sphere/` (core code), `tests/`.
- No “frontend” — it’s a CLI tool. Everything lives inside the package.
- Look for plugin-style modules inside `src/m2_fake_sphere` for different output connectors and LLM integration.

##### Key points from README
- **Core engine** that ingests a CLI command and a user config, orchestrates one or more “targets” (DBs, APIs, streams, files, agents).
- **Intelligence layer**:
- Scraper (`trafilatura`) + Markdown converter (Microsoft Markitdown) to prepare docs for LLM fine-tuning.
- Synthetic-data backends (Faker/Mimesis templates, SDV probabilistic models, scikit-learn distributions).
- LLM fine-tune pipeline (Phi-2 + QLoRA/Axolotl/PEFT) for mapping user config to generator functions.

1) CLI Interface & Config Parser
2) Orchestrator / Core Engine
3) Intelligence Layer
- Scraper & Markdown Converter
- LLM Fine-tune Pipeline
- Synthetic-Data Model Registry (Faker, SDV, SciKit-learn)
4) Data Generation Modules / Connectors
- Database Adapters (Postgres, MySQL, SQLite, …)
- API Emitters (REST, GraphQL)
- Stream Emitters (Kafka, Pulsar, custom)
- File Writers (CSV, JSON, S3/Datalake)
- Agent Flow Simulator
5) External Systems
- Real or remote databases, API endpoints, message brokers, cloud storage, LLM runtime environment
6) Test Suite (under `tests/`)

- User types `sphere` CLI → Config Parser→ Orchestrator
- Orchestrator → Intelligence Layer (to fetch/generate templates, fine-tune model, pick generator functions)
- Intelligence Layer → Model Registry → Data Generation Modules
- Generation Modules → Connectors → External Systems
- Orchestrator → Tests (for CI / validation)

c. Architectural patterns & principles:
- Layered architecture (Presentation/CLI → Application/Orchestrator → Services/Modules → Infrastructure/Connectors)
- Plugin or adapter pattern for adding new connectors or data backends
- Modular separation of concerns: scraping/LLM vs. pure data generation vs. output delivery
- Dependency injection: orchestrator drives modules based on config, not hard-coded flows

d. Key technologies & libraries (annotate next to each component):
- Scraper: `trafilatura`
- Markdown converter: `markitdown`
- Synthetic data: `Faker`, `Mimesis`, `SDV`, `scikit-learn`
- LLM: `Phi-2` model, `QLoRA` + `Axolotl` (`PEFT`)
- Connectors: Python DB drivers, `requests` or `aiohttp`, Kafka/Pulsar client, `boto3` or equivalent

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Layer
    CLI["CLI Interface & Config Parser"]:::cli
    click CLI "https://github.com/mm-cyberlabs/m2-fake-sphere/blob/main/m2_fake_sphere/src/m2_fake_sphere/__init__.py"

    %% Core Engine
    Orchestrator["Core Engine / Orchestrator"]:::core
    click Orchestrator "https://github.com/mm-cyberlabs/m2-fake-sphere/tree/main/m2_fake_sphere/src/m2_fake_sphere/"

    %% Intelligence Layer
    subgraph "Intelligence Layer"
        direction TB
        Scraper["Scraper & Markdown Converter"]:::intel
        LLM["LLM Fine-tune Pipeline"]:::intel
        Registry["Synthetic-Data Model Registry"]:::intel
    end
    click Scraper "https://github.com/mm-cyberlabs/m2-fake-sphere/tree/main/m2_fake_sphere/src/m2_fake_sphere/"
    click LLM "https://github.com/mm-cyberlabs/m2-fake-sphere/tree/main/m2_fake_sphere/src/m2_fake_sphere/"
    click Registry "https://github.com/mm-cyberlabs/m2-fake-sphere/tree/main/m2_fake_sphere/src/m2_fake_sphere/"

    %% Generation Modules
    subgraph "Data Generation Modules / Connectors"
        direction TB
        DB["Database Adapters"]:::gen
        API["API Emitters"]:::gen
        Streams["Stream Emitters"]:::gen
        Files["File Writers"]:::gen
        Agent["Agent Flow Simulator"]:::gen
    end
    click DB "https://github.com/mm-cyberlabs/m2-fake-sphere/tree/main/m2_fake_sphere/src/m2_fake_sphere/"
    click API "https://github.com/mm-cyberlabs/m2-fake-sphere/tree/main/m2_fake_sphere/src/m2_fake_sphere/"
    click Streams "https://github.com/mm-cyberlabs/m2-fake-sphere/tree/main/m2_fake_sphere/src/m2_fake_sphere/"
    click Files "https://github.com/mm-cyberlabs/m2-fake-sphere/tree/main/m2_fake_sphere/src/m2_fake_sphere/"
    click Agent "https://github.com/mm-cyberlabs/m2-fake-sphere/tree/main/m2_fake_sphere/src/m2_fake_sphere/"

    %% External Systems
    subgraph "External Systems"
        direction TB
        RealDB["Real/Remote Databases"]:::external
        Endpoints["API Endpoints"]:::external
        Brokers["Message Brokers"]:::external
        Storage["Cloud Storage"]:::external
        LLMRuntime["LLM Runtime Environment"]:::external
    end

    %% Test Suite
    Tests["Test Suite"]:::cli
    click Tests "https://github.com/mm-cyberlabs/m2-fake-sphere/tree/main/m2_fake_sphere/tests/"

    %% Documentation & Packaging
    subgraph "Project Docs & Packaging"
        direction TB
        README["README.md"]:::docs
        License["LICENSE"]:::docs
        Assets["assets/"]:::docs
        PyProject["pyproject.toml"]:::docs
    end
    click README "https://github.com/mm-cyberlabs/m2-fake-sphere/blob/main/README.md"
    click License "https://github.com/mm-cyberlabs/m2-fake-sphere/tree/main/LICENSE"
    click Assets "https://github.com/mm-cyberlabs/m2-fake-sphere/tree/main/assets/"
    click PyProject "https://github.com/mm-cyberlabs/m2-fake-sphere/blob/main/m2_fake_sphere/pyproject.toml"

    %% Flows
    CLI -->|initiates| Orchestrator
    Orchestrator -->|calls| Scraper
    Orchestrator -->|calls| LLM
    Orchestrator -->|fetches models| Registry
    Registry -->|provides generators| DB
    Registry -->|provides generators| API
    Registry -->|provides generators| Streams
    Registry -->|provides generators| Files
    Registry -->|provides generators| Agent
    DB -->|writes to| RealDB
    API -->|sends to| Endpoints
    Streams -->|publishes to| Brokers
    Files -->|stores in| Storage
    Agent -->|interacts with| Endpoints
    Orchestrator -->|runs| Tests

    %% Styles
    classDef cli fill:#D0E7FF,stroke:#333,stroke-width:1px;
    classDef core fill:#D0FFD8,stroke:#333,stroke-width:1px;
    classDef intel fill:#ECD0FF,stroke:#333,stroke-width:1px;
    classDef gen fill:#FFE7C0,stroke:#333,stroke-width:1px;
    classDef external fill:#E0E0E0,stroke:#333,stroke-width:1px;
    classDef docs fill:#F0F0F0,stroke:#333,stroke-width:1px;
```
