How microsoft/metered-billing-accelerator works
Main Components
• Client Components: These are external applications or SDKs that generate and send two main types of JSON events (“SubscriptionPurchased” and “UsageReported”) into an Azure EventHub.
• EventHub: Acts as the single source‐of‐truth where all usage events and feedback messages from API submissions are recorded. This component also supports EventHub Capture.
• Aggregator (Processing Engine): A central .NET application that reads events from one or more EventHub partitions. Its responsibilities include:
• Reading and applying the event stream to update internal state.
• Operating as a “pure function” (business logic) to fold events onto state (using reactive extensions).
• Managing snapshots (state persistence) via Azure Blob Storage so processing can resume after interruptions.
• Emitting jobs or “usage to be reported” items.
• Blob Storage: Used both to hold the long-term snapshots (state files) for each partition and as a checkpoint mechanism.
• Azure Marketplace Metering API Integration: A dedicated component (within the aggregator) that periodically reads ready-to-be-submitted usage items (aggregated on an hourly basis) and submits them (in batches) to the external API. Feedback (successes or errors) is then emitted back as events.
• Deployment Artifacts: Bicep templates, ARM templates, and various deploy scripts in the “deploy” directory that provision the infrastructure resources (EventHub, Blob Containers, etc.).
Interactions and Data Flow Between Components
• Usage events flow from the client-side (thin SDK layer) → EventHub → Aggregator.
• The Aggregator reads its last known snapshot from Blob Storage, then sequentially processes all new events in each partition.
• The internal business logic “folds” events over state. As certain thresholds (like “included” usage depletion) are met, it records job entries, which then trigger API submission.
• Once a batch of usage items is ready, a dedicated submodule in the Aggregator submits these to the Azure Marketplace Metering API.
• Feedback responses from the API (including duplicate errors for idempotency) circulate back into EventHub, which the system processes accordingly.
• Periodically, the snapshot manager writes an updated JSON snapshot back into Blob storage to serve as the next starting point.
Architectural Patterns and Design Principles
• Event Sourcing: All state transitions are captured as events in EventHub.
• Reactive Processing: The aggregator uses observables (via Reactive Extensions) to continuously emit updated state.
• Pure Business Logic: The aggregation “folding” function is side-effect free, with side effects segregated into snapshot persistence and external API calls.
• Idempotency: The design handles cases where API submissions might be repeated due to external failures.
• Distributed, resilient, and hosting agnostic: The aggregator can be deployed on VMs, as an Azure Function, a Docker container in ACI or AKS, or even on-premises.
Technologies, Frameworks, and Libraries
• .NET and F# (as seen in several .fsproj files) driving the business logic.
• Azure SDK components for EventHub and Blob Storage.
• Bicep/ARM templates for infrastructure deployment.
• Docker-compose files suggesting potential containerization.
• Reactive Extensions (Rx) for event-stream processing.
Final Recommendations
• Be as detailed as possible by separating out subcomponents within the Aggregator. For example, list the “State Updater” that applies events using reactive streams and the “Submission Manager” that handles batching and error feedback.