How bmoscon/cryptofeed works
Main components
• Feed Handler Core – This is the orchestrator that instantiates feeds, manages subscriptions, and starts the event loop.
• Exchange Modules – Under the “cryptofeed/exchanges” directory you have individual modules/classes that implement connection logic and data translation for each supported exchange (e.g., Binance, Coinbase, Kraken, etc.). Also note the “mixins” which provide shared REST functionality.
• Callbacks & Data Normalization – The package offers a system for user-registered callbacks (like ticker, trades, book updates) that are invoked once the data is processed.
• Connection & Messaging – Components such as “cryptofeed/connection.py” and “cryptofeed/connection_handler.py” are responsible for establishing and managing the actual communication with exchange endpoints.
• Utilities & Helpers – The “cryptofeed/util” and error-handling modules support tasks like book management, performance, timing, and format conversions.
Relationships and interactions
• Each exchange module (e.g., binance.py, coinbase.py) connects directly to the corresponding exchange endpoint (via websockets and/or REST) and processes incoming messages.
• Configuration components (like config.py and config.yaml) ensure that the feed handler and exchange modules use the proper settings. You might represent these as a configuration store that feeds into the system.
• Testing and tooling (e.g., “tests/”, “tools/”, “examples/”) can be loosely represented as supporting verification layers, noting that they validate and demonstrate end-to-end data flow.
Architectural patterns and design principles
• Modular Design – The library is structured with clear separations (feed handler, exchanges, backends, utilities) to facilitate extensibility.
• Callback-Driven Data Processing – Callbacks serve as hooks for client code, so representing these as “listener” components is essential.
• Plug-in Backends – Backends are chosen based on user needs; represent these as optional branches that the main data flow can fork to.
• Abstraction Layers – Note that exchange implementations abstract both websocket and REST interactions and that mixins handle shared functionalities.
Relevant technologies
• Python 3.8+ (as noted in the README)
• Asynchronous programming (likely using asyncio)
• Supported external systems (Redis, MongoDB, PostgreSQL, Kafka, ZeroMQ, etc.) should be labeled next to their respective backend components.
• You may add a “Configuration & Deployment” note that includes files like setup.py, config.yaml, and CI/CD config files from .github/workflows, Travis, etc.
Specific elements
• A central "FeedHandler" module that collects and initiates feeds.
• Branches for each major exchange integration (grouped under an "Exchanges" module) showing how data enters.
• A separate “Backends” branch showing various storage or messaging output options.
• A Configuration component feeding into the Feed Handler.
• Notations for support utilities (book management, error handling) that are used internally.