# terrafloww/rasteret architecture

> Architecture diagram of the GitHub repository terrafloww/rasteret, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/terrafloww/rasteret
- Repository: https://github.com/terrafloww/rasteret

## Overview

##### Main Components

a) Core System Components:
- STAC Interface Layer
- COG Processing Engine
- Cloud Provider Interface
- Local Cache System
- Data Output Handlers

b) External Systems:
- Cloud Storage (AWS S3)
- STAC Endpoints (Earth-search, Landsatlook)
- Local File System

**Client Layer**

**API Layer**

**Processing Layer**

**Storage Layer**

**External Services Layer**

##### Specific Component Details

a) Client Layer:
- Entry points through the main Rasteret class
- Both Collection Creation and Data Retrieval flows

b) API Layer (src/rasteret/):
- Core API components
- Type definitions
- Logging system

c) Processing Layer (src/rasteret/core/):
- Collection processor
- Scene processor
- Utility functions

d) Storage Layer:
- Local cache system
- GeoParquet storage
- COG header cache

e) External Services Layer:
- STAC endpoints
- S3 storage
- Authentication systems

##### Data Flow Representations

- Initial STAC metadata retrieval flow
- COG header caching flow
- Data retrieval and processing flow
- Output generation flow (Xarray/GeoDataFrame)

##### Key Technical Elements to Highlight

- Parallel processing capabilities
- Caching mechanism
- STAC metadata indexing
- COG tile retrieval optimization
- Output format conversions

##### Implementation Details

- How src/rasteret/stac/ handles STAC interactions
- How src/rasteret/cloud.py interfaces with cloud providers
- Demonstrate how src/rasteret/core/ manages processing
- Fetch module's role in data retrieval

##### Performance Optimization Points

Areas where the system implements its key optimizations:
- Single HTTP request per image tile
- Local header caching
- Parallel processing implementation
- STAC metadata indexing

Remember to include version information and any relevant notes about the work-in-progress nature of the library, as mentioned in the README warning.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Styles
    classDef api fill:#9370DB
    classDef core fill:#4169E1
    classDef storage fill:#3CB371
    classDef external fill:#FFA500
    classDef utility fill:#A9A9A9

    %% Client Layer
    subgraph ClientLayer
        Client["Client Application"]
        Examples["Example Workflows"]:::utility
    end
    click Examples "https://github.com/terrafloww/rasteret/tree/main/examples/"

    %% API Layer
    subgraph APILayer
        TypeDefs["Type Definitions"]:::api
        Constants["Constants"]:::api
        Logger["Logging System"]:::api
    end
    click TypeDefs "https://github.com/terrafloww/rasteret/blob/main/src/rasteret/types.py"
    click Constants "https://github.com/terrafloww/rasteret/blob/main/src/rasteret/constants.py"
    click Logger "https://github.com/terrafloww/rasteret/blob/main/src/rasteret/logging.py"

    %% Processing Layer
    subgraph ProcessingLayer
        COGEngine["COG Processing Engine"]:::core
        CollectionProc["Collection Processor"]:::core
        SceneProc["Scene Processor"]:::core
        COGHandler["COG Handler"]:::core
        Utils["Utility Functions"]:::utility
    end
    click COGEngine "https://github.com/terrafloww/rasteret/tree/main/src/rasteret/core/"
    click CollectionProc "https://github.com/terrafloww/rasteret/blob/main/src/rasteret/core/collection.py"
    click SceneProc "https://github.com/terrafloww/rasteret/blob/main/src/rasteret/core/scene.py"
    click COGHandler "https://github.com/terrafloww/rasteret/blob/main/src/rasteret/fetch/cog.py"
    click Utils "https://github.com/terrafloww/rasteret/blob/main/src/rasteret/core/utils.py"

    %% STAC Layer
    subgraph STACLayer
        STACInterface["STAC Interface"]:::api
        STACIndexer["STAC Indexer"]:::core
        STACParser["STAC Parser"]:::core
    end
    click STACInterface "https://github.com/terrafloww/rasteret/tree/main/src/rasteret/stac/"
    click STACIndexer "https://github.com/terrafloww/rasteret/blob/main/src/rasteret/stac/indexer.py"
    click STACParser "https://github.com/terrafloww/rasteret/blob/main/src/rasteret/stac/parser.py"

    %% Storage Layer
    subgraph StorageLayer
        LocalCache["Local Cache System"]:::storage
        HeaderCache["COG Header Cache"]:::storage
        CloudProvider["Cloud Provider Interface"]:::api
    end
    click CloudProvider "https://github.com/terrafloww/rasteret/blob/main/src/rasteret/cloud.py"

    %% External Services Layer
    subgraph ExternalLayer
        S3["AWS S3"]:::external
        STACEndpoints["STAC Endpoints"]:::external
        LocalFS["Local File System"]:::external
    end

    %% Testing Components
    subgraph TestingLayer
        CloudTests["Cloud Provider Tests"]:::utility
        STACTests["STAC Indexer Tests"]:::utility
    end
    click CloudTests "https://github.com/terrafloww/rasteret/blob/main/src/rasteret/tests/test_cloud_provider.py"
    click STACTests "https://github.com/terrafloww/rasteret/blob/main/src/rasteret/tests/test_stac_indexer.py"

    %% Relationships
    Client --> Examples
    Client --> TypeDefs
    Client --> COGEngine
    
    COGEngine --> CollectionProc
    COGEngine --> SceneProc
    COGEngine --> COGHandler
    COGEngine --> Utils
    
    CollectionProc --> STACInterface
    SceneProc --> CloudProvider
    
    STACInterface --> STACIndexer
    STACInterface --> STACParser
    
    STACIndexer --> STACEndpoints
    CloudProvider --> S3
    
    COGHandler --> LocalCache
    COGHandler --> HeaderCache
    
    LocalCache --> LocalFS
    HeaderCache --> LocalFS
    
    CloudTests --> CloudProvider
    STACTests --> STACIndexer

    %% Legend
    subgraph Legend
        API["API Component"]:::api
        Core["Core Component"]:::core
        Store["Storage Component"]:::storage
        Ext["External Service"]:::external
        Util["Utility Component"]:::utility
    end
```
