# victor-antoniassi/coinmarketcap_api_to_duckdb architecture

> Architecture diagram of the GitHub repository victor-antoniassi/coinmarketcap_api_to_duckdb, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/victor-antoniassi/coinmarketcap_api_to_duckdb
- Repository: https://github.com/victor-antoniassi/coinmarketcap_api_to_duckdb

## Overview

##### Main Components

a. External Components:
- CoinMarketCap API (represent as an external service)
- Configuration Files (.dlt/secrets.toml)

b. Core Pipeline Components:
- Data Extraction Module
- Data Loading Module
- DuckDB Database

##### Component Relationships

**[CoinMarketCap API] → [Data Extraction Module] → [Data Loading Module] → [DuckDB Database]**
- ↑                    ↑
- [Secrets Config]    [Error Handling]

##### Specific Design Elements

- Rectangles: Internal processing modules
- Clouds: External APIs

- Blue: External services (CoinMarketCap API)
- Green: Data processing components
- Yellow: Configuration elements
- Purple: Storage (DuckDB)

##### Technical details in notes
- Python 3.9+ runtime
- dlt framework usage
- Cryptocurrency symbols (BTC, ETH, LTC)
- Data points tracked (price, volume, market cap, timestamp)

##### Key Architectural Aspects to Highlight

a. Security Layer:
- How API credentials are managed via .dlt/secrets.toml
- Secure API communication

b. Data Processing:
- Validation steps
- Error handling mechanisms
- Data transformation flow

##### Additional Elements

- Data flow indicators
- Component responsibilities

b. Metadata:
- Pipeline frequency
- Data volume indicators
- System requirements

Remember to:
- Keep the main flow clear and central
- Error handling paths
- Configuration touchpoints
- Security aspects of API key management
- Monitoring/logging points

This design will give a clear picture of how the pipeline works while maintaining the appropriate level of technical detail for a principal software engineer to understand the system's architecture and data flow.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Services
    CMP((("CoinMarketCap API"))):::external
    
    %% Configuration
    subgraph Config ["Configuration Layer"]
        SEC[".dlt/secrets.toml"]:::config
        REQ["requirements.txt"]:::config
        DOC["README.md"]:::docs
    end

    %% Core Pipeline
    subgraph Pipeline ["Data Pipeline"]
        EXT["Data Extraction Module"]:::process
        LOAD["Data Loading Module"]:::process
        ERR{"Error Handling"}:::error
        PIPE["dlt_to_duckdb.py"]:::process
    end

    %% Storage
    DB[(DuckDB Database)]:::storage

    %% Relationships
    CMP --> |"API Requests\n(JSON)"| EXT
    SEC --> |"API Credentials"| EXT
    EXT --> |"Raw Data"| PIPE
    PIPE --> |"Transformed Data"| LOAD
    LOAD --> |"Structured Data"| DB
    EXT & LOAD --> ERR
    ERR --> |"Retry/Log"| Pipeline

    %% Technical Details
    classDef external fill:#2374f7,stroke:#164694,stroke-width:2px
    classDef config fill:#ffd700,stroke:#b8860b,stroke-width:2px
    classDef process fill:#90EE90,stroke:#228B22,stroke-width:2px
    classDef storage fill:#DDA0DD,stroke:#800080,stroke-width:2px
    classDef error fill:#FFB6C1,stroke:#FF69B4,stroke-width:2px
    classDef docs fill:#F5F5F5,stroke:#A9A9A9,stroke-width:2px

    %% Click Events
    click SEC "https://github.com/victor-antoniassi/coinmarketcap_api_to_duckdb/blob/main/.dlt/secrets.toml"
    click PIPE "https://github.com/victor-antoniassi/coinmarketcap_api_to_duckdb/blob/main/dlt_to_duckdb.py"
    click REQ "https://github.com/victor-antoniassi/coinmarketcap_api_to_duckdb/blob/main/requirements.txt"
    click DOC "https://github.com/victor-antoniassi/coinmarketcap_api_to_duckdb/blob/main/README.md"

    %% Legend
    subgraph Legend
        L1((("External API"))):::external
        L2["Configuration"]:::config
        L3["Processing"]:::process
        L4[(Storage)]:::storage
        L5{"Error Handler"}:::error
        L6["Documentation"]:::docs
    end
```
