How department-of-veterans-affairs/breakers works
Main Components
a. Application (Ruby code using Faraday)
b. Faraday HTTP Client
c. Breakers Middleware (the gem itself)
• Breakers::Client
• Breakers::Service
• Breakers::Outage / OutageException
• Breakers::UptimeMiddleware
• Plugins & Logger (optional extension points)
d. Redis (data store for metrics & outages)
e. Downstream Backend Service (the actual HTTP service being called)
Relationships and Data Flows
• Request Flow (HTTP request path):
Application → Faraday → Breakers Middleware → (If service up) → Backend Service → Breakers → Application
• Metrics Flow (to Redis):
Breakers Middleware intercepts response or error → records success or error counters in Redis
• Outage Detection & Storage:
Breakers::Client periodically examines counts in Redis → if error rate > threshold → creates Outage record → writes to Redis sorted set → may raise Breakers::OutageException
• Recovery Probes:
While in outage, Breakers allows a “probe” request every N seconds → if success → clears outage in Redis → resumes normal flow
• Plugins & Logging Events:
On each error, success, outage begin/end → Breakers invokes plugin hooks & optional Logger
Architectural Patterns and Principles
• Circuit-Breaker Pattern via pluggable Faraday Middleware
• Separation of concerns: request handling vs. metrics store vs. outage logic vs. plugin interface
• Redis used as a time-series store (per-minute keys) and sorted set for outages
• Dependency injection: Services (with request_matcher), Client (with redis_connection, plugins, logger)
Technologies & Frameworks
• Ruby, Faraday (HTTP client library)
• Redis (in-memory key-value store)
• RSpec for tests, Rake, Bundler, RuboCop for linting
• GitHub Actions / Travis CI for CI