How copperwire/fbmc-quality works
Project type and purpose
• This is a Python-based data-analysis library and CLI tool (“fbmc-quality”) for fetching, storing, validating and analyzing flow-based market clearing (FBMC) data from external sources (JAO and ENTSO-E) and internal caches.
• The README confirms it provides “analysis tools to pull data from the Joint Allocation Office, ENTSO-E Transparency and internal resources to investigate the quality of the flow based market clearing output.”
File structure
• Top-level structure:
• Configuration & CI files (.flake8, .pre-commit-config.yaml, pyproject.toml, .github/workflows, .readthedocs.yaml)
• Documentation folder (doc/)
• Core library package (fbmc_quality/)
• Test suite (tests/)
• Core package subdivided into:
• dataframe_schemas: cache database schemas & validation
• datetime_handlers: timezone handling
• entsoe_data: CLI & functions for fetching/store ENTSO-E data
• jao_data: CLI & functions for fetching/analyzing JAO data
• linearisation_analysis: compute linearisation errors and process data
• linearisation_error_app: high-level application entrypoint
• plotting: flow-map visualization
• enums & exceptions: shared definitions
README for additional insights
• Confirms integration with JAO and ENTSO-E
• Mentions “quality of the flow based market clearing output” → emphasis on validation & error analysis
Main components and their relationships
a. External Data Sources
– JAO API
– ENTSO-E Transparency API
b. Local Cache Database
– Uses dataframe_schemas.cache_db to store raw data
c. Ingestion CLI Layer
– jao_store_cli.py
– entsoe_store_cli.py
– Fetch modules (fetch_jao_data.py, fetch_entsoe_data.py)
d. Data Processing & Validation
– dataframe_schemas.schemas.py (validate loaded tables)
– datetime_handlers.handle_timezones
– linearisation_analysis.process_data & compute_functions
e. Analysis & Business Logic
– linearisation_analysis.dataclasses (typed models)
– conservation rules (test_conservation_rules)
f. Application / UI Layer
– linearisation_error_app.basic_app (entrypoint to run full workflow)
– CLI arguments, configuration reading
g. Visualization
– plotting.flow_map (generates figures)
h. Testing
– tests/ (integration & unit tests)
• Flow of data:
1. Ingestion CLI calls fetch_xxx_data → writes to cache DB
2. Processing modules load data from cache DB → apply timezone handling → validate against schemas
3. Analysis modules compute linearisation errors & conservation rules
4. Visualization module produces flow maps or statistical summaries
5. Application CLI orchestrates end-to-end workflow
Important architectural patterns and design principles
• Modular package structure: separation of concerns by domain (ingestion, schemas, processing, analysis, plotting)
• CLI-driven orchestration: each data source has its own entrypoint
• Schema validation layer decoupled from business logic
• Test suite enforces data integrity and rule correctness
• Documentation generation (Sphinx / ReadTheDocs)