How deepblue2333/streamingmachinelearning works
Based on your provided file tree and README, this project is a streaming compute engine designed to support online machine learning algorithms such as OnlineKMeans, OnlineARIMA, and OnlineBinaryClassifier. Its goal is to enable streaming training with customizable machine learning strategies, window-based computations with watermarking, double-stream joining, and scalability across distributed environments.
Main Components
• Streaming Compute Engine Core: Represented by the “StreamingComputeEngine” directory. It includes the engine package (component execution, event dispatching, job management, and web server) and the job package (various streaming jobs such as FraudDetectionJob, Transaction related jobs, etc.).
• Online Machine Learning Algorithms: These exist in both the “OlineAlgo” and “StreamingComputeEngine/src/main/java/algo” directories. They implement the online versions of ARIMA, KMeans, and Binary Classifier.
• API Layer: Found in StreamingComputeEngine’s “api” package, it provides the interfaces for components like Sources, Operators, Windowing strategies, and Model definitions.
• Resource and Presentation Layer: The “resources” directory contains template files (like index.twig) and static assets (CSS under public) which indicate a basic UI or server-rendered views.
• Testing and Documentation: The “Doc” folder contains documentation on OnlineKMeans, and the “src/test” directories for testing the API and job functionality.
Establish Relationships & Interactions
• Data Ingestion: Represent the Source components (e.g., CsvReaderSource) that feed events into the system.
• Core Processing Engine: Show how events move from Sources, through various Operators (WindowingOperator, TrainOperator, etc.) managed by the engine components (ComponentExecutor, EventDispatcher, SourceExecutor, etc.).
• Model Algorithms: Highlight the integration of streaming algorithms (OnlineARIMA, OnlineBinaryClassifier, OnlineKMeans) wherever these are required by processing jobs.
• Job Execution: Connect how jobs are scheduled (JobStarter) and executed using Operator Executors, linking the streaming data processing with user-defined jobs (e.g. FraudDetectionJob, TransactionSource).
Architectural Patterns and Design Principles
• Modular Design: Components are clearly separated (engine, jobs, API, algorithms) which allows for independent development and scalability.
• Extensibility via API: Mark the API components that enable easy customization of machine learning strategies (e.g., using the Builder pattern for defining event fields).
• Separation of Concerns: Illustrate clear divisions among data ingestion, processing, model computation, and web interfacing.
Technology and Framework References
• Java and Maven: Depicted via the pom.xml files, which are present in “OlineAlgo” and “StreamingComputeEngine.”
• External Libraries: Any libraries in use (implied through the API and engine implementations) should be noted.
• Reference to "Grokking Streaming Systems": Indicate that the design takes inspiration from established streaming system patterns.
Elements
• Clearly labeled components such as:
• Data Sources (CsvReaderSource, etc.)
• API/Interface Layer (API classes, WebServer)
• Processing Engine (ComponentExecutor, EventDispatcher, SourceExecutor, OperatorExecutor)
• Machine Learning Algorithms (OnlineARIMA, OnlineBinaryClassifier, OnlineKMeans)
• Job Modules (various job classes including FraudDetectionJob, TransactionEvent, etc.)
• Ingestion Layer (data sources)
• Processing Layer (stream operators, algorithm modules, engine)
• Output/Visualization/APIs (web server and client resources)
Happy diagramming!