How mm-cyberlabs/m2-fake-sphere works
Project type and purpose
• This is a Python-based, CLI-first synthetic data engine (“Fake Sphere”) whose goal is to generate realistic test data across databases, APIs, streams, files, and agent flows without using real data.
• It’s packaged as a library/CLI tool (m2_fake_sphere), includes scraper/Markdown converter, multiple data-generation backends (Faker, SDV, scikit-learn), and an LLM tie-breaker fine-tune pipeline (Phi-2 + QLoRA/Axolotl).
File structure
• Top-level:
• m2_fake_sphere/ is the Python package: pyproject.toml, src/m2_fake_sphere/ (core code), tests/.
• No “frontend” — it’s a CLI tool. Everything lives inside the package.
• Look for plugin-style modules inside src/m2_fake_sphere for different output connectors and LLM integration.
Key points from README
• Core engine that ingests a CLI command and a user config, orchestrates one or more “targets” (DBs, APIs, streams, files, agents).
• Intelligence layer:
• Scraper (trafilatura) + Markdown converter (Microsoft Markitdown) to prepare docs for LLM fine-tuning.
• Synthetic-data backends (Faker/Mimesis templates, SDV probabilistic models, scikit-learn distributions).
• LLM fine-tune pipeline (Phi-2 + QLoRA/Axolotl/PEFT) for mapping user config to generator functions.
1) CLI Interface & Config Parser
2) Orchestrator / Core Engine
3) Intelligence Layer
• Scraper & Markdown Converter
• LLM Fine-tune Pipeline
• Synthetic-Data Model Registry (Faker, SDV, SciKit-learn)
4) Data Generation Modules / Connectors
• Database Adapters (Postgres, MySQL, SQLite, …)
• API Emitters (REST, GraphQL)
• Stream Emitters (Kafka, Pulsar, custom)
• File Writers (CSV, JSON, S3/Datalake)
• Agent Flow Simulator
5) External Systems
• Real or remote databases, API endpoints, message brokers, cloud storage, LLM runtime environment
6) Test Suite (under tests/)
• User types sphere CLI → Config Parser→ Orchestrator
• Orchestrator → Intelligence Layer (to fetch/generate templates, fine-tune model, pick generator functions)
• Intelligence Layer → Model Registry → Data Generation Modules
• Generation Modules → Connectors → External Systems
• Orchestrator → Tests (for CI / validation)
c. Architectural patterns & principles:
• Layered architecture (Presentation/CLI → Application/Orchestrator → Services/Modules → Infrastructure/Connectors)
• Plugin or adapter pattern for adding new connectors or data backends
• Modular separation of concerns: scraping/LLM vs. pure data generation vs. output delivery
• Dependency injection: orchestrator drives modules based on config, not hard-coded flows
d. Key technologies & libraries (annotate next to each component):
• Scraper: trafilatura
• Markdown converter: markitdown
• Synthetic data: Faker, Mimesis, SDV, scikit-learn
• LLM: Phi-2 model, QLoRA + Axolotl (PEFT)
• Connectors: Python DB drivers, requests or aiohttp, Kafka/Pulsar client, boto3 or equivalent