# tcec-chess/tcecgames architecture

> Architecture diagram of the GitHub repository tcec-chess/tcecgames, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tcec-chess/tcecgames
- Repository: https://github.com/tcec-chess/tcecgames

## Overview

##### Main Components

a. Data Sources Layer
- TCEC Website (tcec-chess.com)
- Game List JSON Source
- Raw PGN Files

b. Processing Layer
- PGN Processing Scripts
- Classification System
- Compactification Pipeline
- Event Tag Normalization

c. Storage Layer
- Master Archive Repository
- Release Packages
- Version Control (Git)

d. Distribution Layer
- GitHub Releases
- Full Format Distribution
- Compacted Format Distribution

##### Component Relationships

- Data flow from TCEC website to local processing
- Processing pipeline steps
- Distribution paths
- Version control interactions

##### Key Processes to Highlight

- Game list fetching and updating
- PGN processing pipeline
- Release package generation
- Testing procedures

##### Specific Implementation Details

Notes for:
- Python scripts (classifyfrc.py, compactifypgn.py, etc.)
- AWK processing scripts
- Makefile automation
- License handling

**[Data Sources]**
- ↓
**[Processing Pipeline]**
- ↓
**[Storage Systems]**
- ↓
**[Distribution Channels]**

Important Elements to Include:

##### Automation Flow
- How the Makefile orchestrates the entire process
- Automated testing and verification steps

##### Data Transformation
- Conversion from raw PGNs to processed formats
- Parallel processing capabilities (-j flag usage)

##### Quality Control
- Testing and verification steps
- Event classification validation

##### Version Control
- Git repository interaction points
- Release package generation flow

This system design should emphasize the project's focus on maintaining a reliable, well-organized chess game archive with proper processing, verification, and distribution mechanisms.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Define styles
    classDef source fill:#a8d1f0,stroke:#333
    classDef processing fill:#90EE90,stroke:#333
    classDef storage fill:#FFE5B4,stroke:#333
    classDef distribution fill:#FFA07A,stroke:#333
    classDef infrastructure fill:#D3D3D3,stroke:#333

    %% Data Sources Layer
    subgraph DataSources
        TCEC["TCEC Website"]:::source
        GameList["Game List JSON"]:::source
        RawPGN["Raw PGN Files"]:::source
    end

    %% Processing Layer
    subgraph ProcessingPipeline
        FetchScript["Data Fetching Script"]:::processing
        GameListProc["Game List Processor"]:::processing
        PGNProcess["PGN Processing Pipeline"]:::processing
        Classify["Classification System"]:::processing
        EventNorm["Event Tag Normalization"]:::processing
        Compact["Compactification Pipeline"]:::processing
        Consistency["Event Tag Consistency Check"]:::processing
    end

    %% Storage Layer
    subgraph StorageSystem
        MasterArchive["Master Archive Repository"]:::storage
        GitVC["Version Control (Git)"]:::infrastructure
    end

    %% Distribution Layer
    subgraph Distribution
        GHReleases["GitHub Releases"]:::distribution
        FullFormat["Full Format Distribution"]:::distribution
        CompactFormat["Compacted Format Distribution"]:::distribution
    end

    %% Relationships
    TCEC --> FetchScript
    GameList --> GameListProc
    RawPGN --> PGNProcess
    
    FetchScript --> GameListProc
    GameListProc --> PGNProcess
    PGNProcess --> Classify
    Classify --> EventNorm
    EventNorm --> Compact
    Compact --> Consistency
    
    Consistency --> MasterArchive
    MasterArchive --> GitVC
    GitVC --> GHReleases
    GHReleases --> FullFormat
    GHReleases --> CompactFormat

    %% Click events for component mapping
    click GameList "https://github.com/TCEC-Chess/tcecgames/blob/main/master-archive/gamelist.json"
    click RawPGN "https://github.com/TCEC-Chess/tcecgames/blob/main/master-archive/*.pgn"
    click Compact "https://github.com/TCEC-Chess/tcecgames/blob/main/scripts/compactifypgn.py"
    click Classify "https://github.com/TCEC-Chess/tcecgames/blob/main/scripts/classifyfrc.py"
    click EventNorm "https://github.com/TCEC-Chess/tcecgames/blob/main/scripts/normalize-pgn-event-tag.awk"
    click FetchScript "https://github.com/TCEC-Chess/tcecgames/blob/main/scripts/fetch-update-gamelist.sh"
    click GameListProc "https://github.com/TCEC-Chess/tcecgames/blob/main/scripts/processgamelistjson.py"
    click MasterArchive "https://github.com/TCEC-Chess/tcecgames/tree/main/master-archive"
    click Consistency "https://github.com/TCEC-Chess/tcecgames/blob/main/scripts/scan-event-tag-consistency.awk"

    %% Legend
    subgraph Legend
        L1["Data Sources"]:::source
        L2["Processing Steps"]:::processing
        L3["Storage Systems"]:::storage
        L4["Distribution Channels"]:::distribution
        L5["Infrastructure"]:::infrastructure
    end
```
