# apache/orc architecture

> Architecture diagram of the GitHub repository apache/orc, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apache/orc
- Repository: https://github.com/apache/orc

## Overview

##### Main Components

a. Core Components:
- C++ Library (Reader/Writer)
- Java Library (Reader/Writer)
- File Format Layer
- Tools Layer

b. Key Processing Elements:
- Column Readers/Writers
- Compression Codecs
- Memory Management
- Schema Evolution
- Encryption System
- SIMD Optimizations (AVX512)

##### Architectural Layers

**[ORC File Structure]**
- File Metadata
- Stripes (Data Blocks)
- Index Data
- Row Data
- Stripe Footer
- File Footer

Second Layer (Processing):
**[Reader/Writer Components]**
- Type System
- Column Processors
- Compression
- Memory Management
- Statistics

##### Data Flow Representation

- Data reading path: File → Decompression → Column Readers → Memory
- Data writing path: Memory → Column Writers → Compression → File
- Index usage: Query → Predicate Pushdown → Index → Data Access

##### Integration Points

Connections with:
- Hadoop ecosystem
- MapReduce integration
- External tools
- Client applications

##### Specific Implementation Details

Specialized components:
**[Optimization Layer]**
- BloomFilters
- Index Structures
- SIMD Operations
- Memory Pools

##### Key Relationships to Highlight

- Reader/Writer interaction with file format
- Memory management across components
- Compression pipeline
- Type system implementation
- Statistics collection and usage

##### Performance Features

Optimization mechanisms:
- Predicate pushdown
- Column pruning
- SIMD vectorization
- Compression selection

##### Security Layer

Encryption and security features:
- Encryption keys management
- Data masking
- Access control

This architecture should emphasize ORC's key features: columnar storage, type awareness, predicate pushdown, and high-performance reading capabilities, while showing how these features are implemented across both C++ and Java libraries.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph "File Format Layer"
        FileStruct["ORC File Structure"]
        Metadata["File Metadata"]
        Stripes["Stripes (Data Blocks)"]
        Footer["File Footer"]
        
        FileStruct --> Metadata
        FileStruct --> Stripes
        FileStruct --> Footer
        
        subgraph "Stripe Components"
            IndexData["Index Data"]
            RowData["Row Data"]
            StripeFooter["Stripe Footer"]
            
            Stripes --> IndexData
            Stripes --> RowData
            Stripes --> StripeFooter
        end
    end

    subgraph "Core Processing Layer"
        CppCore["C++ Core Library"]:::cpp
        JavaCore["Java Core Library"]:::java
        
        subgraph "Reader/Writer Components"
            CppReader["C++ Reader"]:::cpp
            CppWriter["C++ Writer"]:::cpp
            JavaReader["Java Reader"]:::java
            JavaWriter["Java Writer"]:::java
            
            CppCore --> CppReader
            CppCore --> CppWriter
            JavaCore --> JavaReader
            JavaCore --> JavaWriter
        end
        
        ColumnProc["Column Processing"]:::processing
        TypeSystem["Type System"]:::core
        CompSystem["Compression System"]:::processing
        MemManagement["Memory Management"]:::core
        SchemaEvol["Schema Evolution"]:::core
        Stats["Statistics"]:::processing
    end

    subgraph "Optimization Layer"
        BloomFilters["Bloom Filters"]:::opt
        IndexStructs["Index Structures"]:::opt
        SIMD["SIMD Operations"]:::opt
        MemPools["Memory Pools"]:::opt
    end

    subgraph "Tools Layer"
        FileScan["File Scanner"]:::tools
        FileStats["File Statistics"]:::tools
        Tools["ORC Tools"]:::tools
    end

    %% Core Processing Relationships
    CppReader --> ColumnProc
    CppWriter --> ColumnProc
    JavaReader --> ColumnProc
    JavaWriter --> ColumnProc
    
    ColumnProc --> CompSystem
    ColumnProc --> TypeSystem
    ColumnProc --> MemManagement
    ColumnProc --> Stats
    
    %% Optimization Relationships
    MemManagement --> MemPools
    IndexData --> BloomFilters
    IndexData --> IndexStructs
    ColumnProc --> SIMD

    %% File Format Relationships
    CppReader --> FileStruct
    CppWriter --> FileStruct
    JavaReader --> FileStruct
    JavaWriter --> FileStruct

    %% Tools Relationships
    FileScan --> CppCore
    FileStats --> CppCore
    Tools --> JavaCore

    %% Click Events
    click CppReader "https://github.com/apache/orc/blob/main/c++/include/orc/Reader.hh"
    click CppWriter "https://github.com/apache/orc/blob/main/c++/include/orc/Writer.hh"
    click JavaReader "https://github.com/apache/orc/blob/main/java/core/src/java/org/apache/orc/Reader.java"
    click JavaWriter "https://github.com/apache/orc/blob/main/java/core/src/java/org/apache/orc/Writer.java"
    click ColumnProc "https://github.com/apache/orc/blob/main/c++/src/ColumnReader.cc"
    click MemManagement "https://github.com/apache/orc/blob/main/c++/include/orc/MemoryPool.hh"
    click SchemaEvol "https://github.com/apache/orc/blob/main/c++/src/SchemaEvolution.cc"
    click CompSystem "https://github.com/apache/orc/blob/main/c++/src/Compression.cc"
    click SIMD "https://github.com/apache/orc/blob/main/c++/src/BitUnpackerAvx512.hh"
    click FileScan "https://github.com/apache/orc/blob/main/tools/src/FileScan.cc"
    click FileStats "https://github.com/apache/orc/blob/main/tools/src/FileStatistics.cc"
    click Tools "https://github.com/apache/orc/tree/main/java/tools/src/java/org/apache/orc/tools/"
    click TypeSystem "https://github.com/apache/orc/blob/main/c++/include/orc/Type.hh"
    click Stats "https://github.com/apache/orc/blob/main/c++/include/orc/Statistics.hh"

    %% Styling
    classDef cpp fill:#2374ab
    classDef java fill:#ff7f50
    classDef processing fill:#7fb800
    classDef core fill:#4a4e4d
    classDef opt fill:#9b4dca
    classDef tools fill:#ffa700
```
