# copperwire/fbmc-quality architecture

> Architecture diagram of the GitHub repository copperwire/fbmc-quality, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/copperwire/fbmc-quality
- Repository: https://github.com/copperwire/fbmc-quality

## Overview

##### Project type and purpose
- This is a Python-based data-analysis library and CLI tool (“fbmc-quality”) for fetching, storing, validating and analyzing flow-based market clearing (FBMC) data from external sources (JAO and ENTSO-E) and internal caches.
- The README confirms it provides “analysis tools to pull data from the Joint Allocation Office, ENTSO-E Transparency and internal resources to investigate the quality of the flow based market clearing output.”

##### File structure
- Top-level structure:
- Configuration & CI files (.flake8, .pre-commit-config.yaml, pyproject.toml, .github/workflows, .readthedocs.yaml)
- Documentation folder (doc/)
- Core library package (fbmc_quality/)
- Test suite (tests/)
- Core package subdivided into:
- dataframe_schemas: cache database schemas & validation
- datetime_handlers: timezone handling
- entsoe_data: CLI & functions for fetching/store ENTSO-E data
- jao_data: CLI & functions for fetching/analyzing JAO data
- linearisation_analysis: compute linearisation errors and process data
- linearisation_error_app: high-level application entrypoint
- plotting: flow-map visualization
- enums & exceptions: shared definitions

##### README for additional insights
- Confirms integration with JAO and ENTSO-E
- Mentions “quality of the flow based market clearing output” → emphasis on validation & error analysis

##### Main components and their relationships
a. External Data Sources
– JAO API
– ENTSO-E Transparency API
b. Local Cache Database
– Uses dataframe_schemas.cache_db to store raw data
c. Ingestion CLI Layer
– jao_store_cli.py
– entsoe_store_cli.py
– Fetch modules (fetch_jao_data.py, fetch_entsoe_data.py)
d. Data Processing & Validation
– dataframe_schemas.schemas.py (validate loaded tables)
– datetime_handlers.handle_timezones
– linearisation_analysis.process_data & compute_functions
e. Analysis & Business Logic
– linearisation_analysis.dataclasses (typed models)
– conservation rules (test_conservation_rules)
f. Application / UI Layer
– linearisation_error_app.basic_app (entrypoint to run full workflow)
– CLI arguments, configuration reading
g. Visualization
– plotting.flow_map (generates figures)
h. Testing
– tests/ (integration & unit tests)

- Flow of data:
1. Ingestion CLI calls fetch_xxx_data → writes to cache DB
2. Processing modules load data from cache DB → apply timezone handling → validate against schemas
3. Analysis modules compute linearisation errors & conservation rules
4. Visualization module produces flow maps or statistical summaries
5. Application CLI orchestrates end-to-end workflow

##### Important architectural patterns and design principles
- Modular package structure: separation of concerns by domain (ingestion, schemas, processing, analysis, plotting)
- CLI-driven orchestration: each data source has its own entrypoint
- Schema validation layer decoupled from business logic
- Test suite enforces data integrity and rule correctness
- Documentation generation (Sphinx / ReadTheDocs)

## Diagram (Mermaid)

```mermaid
flowchart TB
  %% External APIs
  subgraph "External APIs"
    direction TB
    JAOAPI["JAO API"]:::external
    ENTSOEEAPI["ENTSO-E API"]:::external
  end

  %% CLI Ingestion Layer
  subgraph "Ingestion CLI" 
    direction TB
    JAOCLI["jao_store_cli"]:::ui
    FetchJAO["fetch_jao_data"]:::compute
    EntsoeCLI["entsoe_store_cli"]:::ui
    FetchEntsoe["fetch_entsoe_data"]:::compute
  end

  %% Storage
  subgraph "Storage"
    direction TB
    CacheDB["Local Cache DB"]:::storage
  end

  %% Validation & Preprocessing
  subgraph "Validation & Preprocessing"
    direction TB
    SchemaVal["schemas"]:::support
    Timezone["handle_timezones"]:::support
  end

  %% Processing & Analysis
  subgraph "Processing & Analysis"
    direction TB
    ProcessData["process_data"]:::compute
    Compute["compute_functions"]:::compute
    Models["dataclasses"]:::support
    Conservation["Conservation Rules"]:::compute
  end

  %% Visualization & App
  subgraph "Visualization & App"
    direction TB
    FlowMap["flow_map"]:::compute
    App["basic_app"]:::ui
  end

  %% Support Artifacts
  subgraph "Support Artifacts"
    direction TB
    Enums["bidding_zones"]:::support
    Exceptions["fbmc_exceptions"]:::support
  end

  %% Tests & Documentation
  subgraph "Tests & Documentation"
    direction TB
    Tests["Tests Suite"]:::support
    Docs["Documentation"]:::support
  end

  %% Data Flow Connections
  JAOAPI -->|"fetch_jao_data"| FetchJAO
  ENTSOEEAPI -->|"fetch_entsoe_data"| FetchEntsoe
  JAOCLI -->|"calls"| FetchJAO
  EntsoeCLI -->|"calls"| FetchEntsoe
  FetchJAO -->|"writes"| CacheDB
  FetchEntsoe -->|"writes"| CacheDB
  CacheDB -->|"validate"| SchemaVal
  SchemaVal -->|"timezone handling"| Timezone
  Timezone -->|"preprocess"| ProcessData
  ProcessData -->|"compute_errors"| Compute
  Compute -->|"generate models"| Models
  Compute -->|"check conservation"| Conservation
  Models -->|"analysis input"| FlowMap
  Conservation -->|"analysis input"| FlowMap

  App -->|"orchestrates"| JAOCLI
  App -->|"orchestrates"| EntsoeCLI
  App -->|"loads"| CacheDB
  App -->|"runs"| SchemaVal
  App -->|"runs"| ProcessData
  App -->|"runs"| Compute
  App -->|"visualizes"| FlowMap

  Enums -.-> Models
  Exceptions -.-> SchemaVal
  Exceptions -.-> Timezone
  Exceptions -.-> ProcessData
  Exceptions -.-> Compute

  Tests -->|"test validation & rules"| SchemaVal
  Tests -->|"test processing"| ProcessData
  Docs -->|"generate docs"| SchemaVal
  Docs -->|"generate docs"| ProcessData
  Docs -->|"generate docs"| FlowMap

  %% Click Events
  click FetchJAO "https://github.com/copperwire/fbmc-quality/blob/main/fbmc_quality/jao_data/fetch_jao_data.py"
  click JAOCLI "https://github.com/copperwire/fbmc-quality/blob/main/fbmc_quality/jao_data/jao_store_cli.py"
  click EntsoeCLI "https://github.com/copperwire/fbmc-quality/blob/main/fbmc_quality/entsoe_data/entsoe_store_cli.py"
  click FetchEntsoe "https://github.com/copperwire/fbmc-quality/blob/main/fbmc_quality/entsoe_data/fetch_entsoe_data.py"
  click CacheDB "https://github.com/copperwire/fbmc-quality/blob/main/fbmc_quality/dataframe_schemas/cache_db/cache_db_functions.py"
  click SchemaVal "https://github.com/copperwire/fbmc-quality/blob/main/fbmc_quality/dataframe_schemas/schemas.py"
  click Timezone "https://github.com/copperwire/fbmc-quality/blob/main/fbmc_quality/datetime_handlers/handle_timezones.py"
  click ProcessData "https://github.com/copperwire/fbmc-quality/blob/main/fbmc_quality/linearisation_analysis/process_data.py"
  click Compute "https://github.com/copperwire/fbmc-quality/blob/main/fbmc_quality/linearisation_analysis/compute_functions.py"
  click Models "https://github.com/copperwire/fbmc-quality/blob/main/fbmc_quality/linearisation_analysis/dataclasses.py"
  click App "https://github.com/copperwire/fbmc-quality/blob/main/fbmc_quality/linearisation_error_app/basic_app.py"
  click FlowMap "https://github.com/copperwire/fbmc-quality/blob/main/fbmc_quality/plotting/flow_map.py"
  click Enums "https://github.com/copperwire/fbmc-quality/blob/main/fbmc_quality/enums/bidding_zones.py"
  click Exceptions "https://github.com/copperwire/fbmc-quality/blob/main/fbmc_quality/exceptions/fbmc_exceptions.py"
  click Tests "https://github.com/copperwire/fbmc-quality/blob/main/tests/test_conservation_rules.py"
  click Docs "https://github.com/copperwire/fbmc-quality/tree/main/doc/"

  %% Styles
  classDef external fill:#cce5ff,stroke:#3399ff,color:#000000;
  classDef storage fill:#d4edda,stroke:#28a745,color:#000000;
  classDef compute fill:#fff3cd,stroke:#ffc107,color:#000000;
  classDef ui fill:#ffe5b4,stroke:#ff8c00,color:#000000;
  classDef support fill:#e2e3e5,stroke:#6c757d,color:#000000;
```
