# olacabs/fabric architecture

> Architecture diagram of the GitHub repository olacabs/fabric, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/olacabs/fabric
- Repository: https://github.com/olacabs/fabric

## Overview

##### Project type and purpose
- This is a Java-based, multi-module, real-time stream processing framework (“Fabric”) with:
- A management service (fabric-manager) for registering and tracking topologies
- An executor (fabric-executor) that launches computations on Mesos/Marathon using Docker
- A core compute engine (fabric-compute-framework) implementing an in-JVM event pipeline
- Pluggable components (fabric-components) such as Kafka sources, filters, HTTP/Kafka writers
- A shared model (fabric-model) and common utilities (fabric-common)

##### File structure
- Top-level aggregator POM ties together submodules:
- fabric-common – shared utilities (DNS resolution, property-reading, exception helpers)
- fabric-model – domain objects (ComponentMetadata, ComputationSpec, EventSet, Processor, Source)
- fabric-components – user-provided processors and sources (JSON filter, Kafka source/writer, HTTP writer)
- fabric-compute-framework – core pipeline, comms channels, builder, loader, linker
- fabric-executor – pulls JARs, configures and runs computation in Docker on Mesos
- fabric-manager – REST API service, DAO layer, service registry, OpenTSDB metrics, config via Dropwizard/YAML
- fabric-examples – sample topology code (RandomSentenceSource, SplitterProcessor, WordCountProcessor)
- fabric-processor-archetype – Maven archetype for new processors

##### README for additional insights
- Purpose: high-throughput (>150K eps/partition), reliable, replayable stream processing
- Architecture highlights:
- Mesos & Marathon for cluster management
- Docker for component isolation
- Dynamic topology assembly from Artifactory
- Core pipeline uses Disruptor or BlockingQueue channels
- Supports event-driven (Stream) and timer-driven (Scheduled) processors
- REST API for registration and control

1. fabric-manager (REST API service)
2. Configuration store/Database (MySQL/Postgres for DAO)
3. Service Discovery / Registry (built into manager or external, e.g., Zookeeper)
4. Artifactory (Jar repository)
5. Mesos/Marathon Cluster
6. Docker Container(s) for executor
7. fabric-executor (inside Docker)
8. fabric-compute-framework (in-JVM pipeline inside executor)
9. fabric-components (loaded dynamically via JAR)
10. External data sources/sinks: Kafka cluster, HTTP endpoints, other queues
- Operator → Manager API (register new computation spec)
- Manager ↔ Database (store/retrieve computation and component metadata)
- Manager → Service Registry (announce services)
- Executor → Manager API (poll for new topologies)
- Executor → Artifactory (fetch component JARs)
- Executor → Mesos/Marathon (launch Docker tasks)
- Docker Container runs fabric-executor
- Executor → Compute Framework API (initialize pipeline)
- Compute Framework ↔ Components (load via reflection, call initialize, consume)
- Compute Framework → External sources (KafkaSource), external sinks (HttpWriter, KafkaWriter)
- Compute Framework inter-stage comms via Disruptor/BlockingQueue
c. Architectural patterns and design principles
- Multi-module Maven (separation of concerns: model, common, manager, executor, compute, components)
- Plugin architecture (components discovered via annotations and loaded at runtime)
- Pipeline pattern (serial in-JVM pipeline per partition)
- Microservice for management API
- Containerization and orchestration (Docker + Mesos/Marathon)
- Externalized configuration (YAML, properties, environment)
d. Relevant technologies, frameworks, libraries
- Java 8, Maven, Jackson (JSON serialization), Disruptor (optional channel), BlockingQueueCommsChannel
- Mesos, Marathon, Docker
- Kafka, Zookeeper
- Dropwizard or Jersey for REST API in manager
- JDBC / MyBatis / JPA for DAO (in manager)
- Artifactory for JAR hosting
- OpenTSDB for metrics

##### Detail level
- Sub-components within compute framework:
- PipelineStage → CommsChannel → ProcessorBase
- Where components are loaded (JarDownloader → ClassLoader)
- Where metrics are emitted (OpenTSDB reporter in manager, or within executor)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Control Plane
    subgraph "Control Plane"
        direction TB
        Operator["Operator"]:::external
        Manager["fabric-manager"]:::core
        DB["Database<br/>(MySQL/Postgres)"]:::external
        Registry["Service Registry<br/>(Zookeeper)"]:::external
        Artifactory["Artifactory"]:::external
        Mesos["Mesos/Marathon"]:::external
        ExecutorContainer["Docker Container"]:::external
        Executor["fabric-executor"]:::core
    end

    %% Data Plane
    subgraph "Data Plane"
        direction TB
        Compute["fabric-compute-framework"]:::runtime
        Components["fabric-components"]:::runtime
        Kafka["Kafka Cluster"]:::external
        HTTP["HTTP Endpoint"]:::external
    end

    %% High-level modules
    AggregatorPOM["Top-level POM"]:::core

    %% Control Plane Relationships
    Operator -->|"HTTP API (REST)"| Manager
    Manager -->|"JDBC"| DB
    Manager -->|"announce"| Registry
    Executor -->|"poll API (REST)"| Manager
    Executor -->|"fetch JARs"| Artifactory
    Executor -->|"launch tasks"| Mesos
    Mesos -->|runs| ExecutorContainer
    ExecutorContainer -->|runs| Executor

    %% Data Plane Relationships
    Executor -->|"init pipeline"| Compute
    Compute -->|"load plugins"| Components
    Compute -->|"consume/produce events"| Kafka
    Compute -->|"HTTP calls"| HTTP
    Compute -->|"inter-stage comms"| Compute

    %% Control -> Data
    ExecutorContainer -->|"data processing"| Compute

    %% Modules
    Manager --> AggregatorPOM
    Manager --> Executor
    Manager --> Compute

    %% Click Events for modules
    click AggregatorPOM "https://github.com/olacabs/fabric/blob/develop/pom.xml"
    click Manager "https://github.com/olacabs/fabric/tree/develop/fabric-manager/"
    click Executor "https://github.com/olacabs/fabric/tree/develop/fabric-executor/"
    click Compute "https://github.com/olacabs/fabric/tree/develop/fabric-compute-framework/"
    click Components "https://github.com/olacabs/fabric/tree/develop/fabric-components/"
    click ServiceCommon "https://github.com/olacabs/fabric/tree/develop/fabric-common/"
    click Model "https://github.com/olacabs/fabric/tree/develop/fabric-model/"
    click Examples "https://github.com/olacabs/fabric/tree/develop/fabric-examples/"
    click Archetype "https://github.com/olacabs/fabric/tree/develop/fabric-processor-archetype/"

    %% Styles
    classDef core fill:#cce5ff,stroke:#2b6cb0,color:#2b6cb0;
    classDef runtime fill:#d4f5d4,stroke:#2f855a,color:#2f855a;
    classDef external fill:#ffe5b4,stroke:#dd6b20,color:#dd6b20;
```
