How olacabs/fabric works
Project type and purpose
• This is a Java-based, multi-module, real-time stream processing framework (“Fabric”) with:
• A management service (fabric-manager) for registering and tracking topologies
• An executor (fabric-executor) that launches computations on Mesos/Marathon using Docker
• A core compute engine (fabric-compute-framework) implementing an in-JVM event pipeline
• Pluggable components (fabric-components) such as Kafka sources, filters, HTTP/Kafka writers
• A shared model (fabric-model) and common utilities (fabric-common)
File structure
• Top-level aggregator POM ties together submodules:
• fabric-common – shared utilities (DNS resolution, property-reading, exception helpers)
• fabric-model – domain objects (ComponentMetadata, ComputationSpec, EventSet, Processor, Source)
• fabric-components – user-provided processors and sources (JSON filter, Kafka source/writer, HTTP writer)
• fabric-compute-framework – core pipeline, comms channels, builder, loader, linker
• fabric-executor – pulls JARs, configures and runs computation in Docker on Mesos
• fabric-manager – REST API service, DAO layer, service registry, OpenTSDB metrics, config via Dropwizard/YAML
• fabric-examples – sample topology code (RandomSentenceSource, SplitterProcessor, WordCountProcessor)
• fabric-processor-archetype – Maven archetype for new processors
README for additional insights
• Purpose: high-throughput (>150K eps/partition), reliable, replayable stream processing
• Architecture highlights:
• Mesos & Marathon for cluster management
• Docker for component isolation
• Dynamic topology assembly from Artifactory
• Core pipeline uses Disruptor or BlockingQueue channels
• Supports event-driven (Stream) and timer-driven (Scheduled) processors
• REST API for registration and control
1. fabric-manager (REST API service)
2. Configuration store/Database (MySQL/Postgres for DAO)
3. Service Discovery / Registry (built into manager or external, e.g., Zookeeper)
4. Artifactory (Jar repository)
5. Mesos/Marathon Cluster
6. Docker Container(s) for executor
7. fabric-executor (inside Docker)
8. fabric-compute-framework (in-JVM pipeline inside executor)
9. fabric-components (loaded dynamically via JAR)
10. External data sources/sinks: Kafka cluster, HTTP endpoints, other queues
• Operator → Manager API (register new computation spec)
• Manager ↔ Database (store/retrieve computation and component metadata)
• Manager → Service Registry (announce services)
• Executor → Manager API (poll for new topologies)
• Executor → Artifactory (fetch component JARs)
• Executor → Mesos/Marathon (launch Docker tasks)
• Docker Container runs fabric-executor
• Executor → Compute Framework API (initialize pipeline)
• Compute Framework ↔ Components (load via reflection, call initialize, consume)
• Compute Framework → External sources (KafkaSource), external sinks (HttpWriter, KafkaWriter)
• Compute Framework inter-stage comms via Disruptor/BlockingQueue
c. Architectural patterns and design principles
• Multi-module Maven (separation of concerns: model, common, manager, executor, compute, components)
• Plugin architecture (components discovered via annotations and loaded at runtime)
• Pipeline pattern (serial in-JVM pipeline per partition)
• Microservice for management API
• Containerization and orchestration (Docker + Mesos/Marathon)
• Externalized configuration (YAML, properties, environment)
d. Relevant technologies, frameworks, libraries
• Java 8, Maven, Jackson (JSON serialization), Disruptor (optional channel), BlockingQueueCommsChannel
• Mesos, Marathon, Docker
• Kafka, Zookeeper
• Dropwizard or Jersey for REST API in manager
• JDBC / MyBatis / JPA for DAO (in manager)
• Artifactory for JAR hosting
• OpenTSDB for metrics
Detail level
• Sub-components within compute framework:
• PipelineStage → CommsChannel → ProcessorBase
• Where components are loaded (JarDownloader → ClassLoader)
• Where metrics are emitted (OpenTSDB reporter in manager, or within executor)