# megvii-research/megfile architecture

> Architecture diagram of the GitHub repository megvii-research/megfile, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/megvii-research/megfile
- Repository: https://github.com/megvii-research/megfile

## Overview

##### Project Type and Purpose
- megfile is an open-source Python library (with an optional CLI) that provides a unified, protocol-agnostic file-system interface.
- Purpose: let users perform file operations (open, glob, stat, sync, etc.) transparently across local FS, S3, HDFS, SFTP, HTTP, and stdio backends.

##### File Structure
- Top-level Python package “megfile” contains:
– cli.py (command‐line interface)
– config.py (configuration loader/resolver)
– interfaces.py (core API definitions)
– fs.py / fs_path.py, s3.py / s3_path.py, http.py / http_path.py, hdfs.py / hdfs_path.py, sftp.py / sftp_path.py, stdio.py / stdio_path.py (protocol modules)
– lib/ (helper implementations: prefetch readers, buffering, caching, glob, joinpath, etc.)
– utils/ (small utilities like mutex)
– version.py, errors.py
- README describes a functional interface (smart_open, smart_glob, etc.) and a Path-oriented interface (SmartPath).

##### README for Insights
- Lists supported protocols, shows both functional and SmartPath interfaces, and details configuration (env vars, config files, aliases).
- Emphasizes multi-threaded s3 readers/writers, zsh-style glob support, and full type hints.

##### Main Components
a. **User Entry Points**
- Python API: `smart_open()`, `smart_glob()`, `SmartPath`
- CLI: `megfile` command
b. **Configuration Resolver**
- Reads environment variables / config files / aliases (in `config.py`)
- Feeds per‐protocol credentials & endpoints into the core
c. **Core Dispatcher / Interfaces**
- `interfaces.py` defines abstract FileSystem and Path interfaces
- The dispatcher (“factory”) chooses the correct protocol module based on URL scheme
d. **Protocol Modules (Plugins)**
For each protocol (fs, s3, hdfs, sftp, http, stdio):
– Path class (e.g. `S3Path`)
– Functional interface (e.g. `smart_open` delegates to `s3.py`)
– Underlying handler implementations in `lib/` (prefetch readers, buffers, cache readers, etc.)
e. **Utility Library**
- Shared code in `megfile/lib/` (glob, joinpath, compare, compatibility shims)
- `utils/` (e.g. mutex)
f. **External Services**
- Local File System
- S3 endpoints
- HDFS NameNode / DataNode
- SFTP server
- HTTP server
- STDIO (stdin/stdout)

##### Relationships & Data Flow
1. **User/CLI** → calls → **Configuration Resolver** → provides credentials & settings
2. **User/CLI** → calls → **Functional API** (`smart_*` functions) or **SmartPath**
3. API layer → **Dispatcher** → instantiates correct **Protocol Module**
4. Protocol Module → uses **lib/** handlers (prefetch readers, buffered writers) + **utils/**
5. Handler threads / I/O → talk to **External Service** (e.g. S3 over HTTP, FS via os calls)
6. Responses propagate back up the chain to the user.

##### Architectural Patterns & Design Principles
- **Factory/Plugin Pattern** for protocol dispatch
- **Adapter Pattern**: Path classes adapt protocol details to a uniform Path interface
- **Strategy Pattern** in lib/ for different reader/writer strategies (buffering, prefetch)
- **Separation of Concerns**: distinct layers for config, API, dispatch, protocol implementation, utilities, and external I/O.

##### Essential Elements
– Clear component **names** (e.g. “S3Path + s3_prefetch_reader”)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Entry Points
    CLI["CLI"]:::entry
    Functional["Functional API\n(smart_* functions)"]:::entry
    SmartPath["SmartPath API"]:::entry

    %% Configuration
    Config["Config Resolver\n(env & config files)"]:::core

    %% Core
    Dispatcher["Dispatcher / interfaces.py"]:::core
    ErrorDef["Errors (errors.py)"]:::core
    Version["Version (version.py)"]:::core
    Pathlike["Path-Like Adapter\n(pathlike.py)"]:::core

    %% Protocol Modules
    subgraph "Protocol Modules"
        FS["FSPath & fs functions"]:::protocol
        S3["S3Path & s3 functions"]:::protocol
        HDFS["HDFSPath & hdfs functions"]:::protocol
        SFTP["SFTPPath & sftp functions"]:::protocol
        HTTP["HTTPPath & http functions"]:::protocol
        STDIO["STDIOPath & stdio functions"]:::protocol
    end

    %% Utilities
    subgraph "Utilities"
        Lib["megfile/lib"]:::util
        Utils["megfile/utils"]:::util
    end

    %% External Services
    subgraph "External Services"
        OSFS["Local File System"]:::external
        S3Svc["S3 Service"]:::external
        HDFSCluster["HDFS Cluster"]:::external
        SFTPServer["SFTP Server"]:::external
        HTTPServer["HTTP Server"]:::external
        STDIOExt["STDIO"]:::external
    end

    %% Connections: Entry to Config & Core
    CLI -->|"resolve credentials"| Config
    Functional -->|"resolve credentials"| Config
    SmartPath -->|"resolve credentials"| Config

    CLI -->|"smart_open()/smart_glob()"| Dispatcher
    Functional -->|"calls smart_*"| Dispatcher
    SmartPath -->|"uses Path interface"| Dispatcher

    Config -->|"provides settings"| Dispatcher

    %% Core internal
    Dispatcher -->|"error handling"| ErrorDef
    Dispatcher -->|"uses adapter"| Pathlike
    Dispatcher -->|"dispatch_by_scheme"| FS
    Dispatcher -->|"dispatch_by_scheme"| S3
    Dispatcher -->|"dispatch_by_scheme"| HDFS
    Dispatcher -->|"dispatch_by_scheme"| SFTP
    Dispatcher -->|"dispatch_by_scheme"| HTTP
    Dispatcher -->|"dispatch_by_scheme"| STDIO

    %% Protocol to Utility
    FS -->|"prefetch_read/write"| Lib
    S3 -->|"prefetch_read/write"| Lib
    HDFS -->|"prefetch_read/write"| Lib
    SFTP -->|"prefetch_read/write"| Lib
    HTTP -->|"prefetch_read/write"| Lib
    STDIO -->|"prefetch_read/write"| Lib

    FS -->|"mutex etc."| Utils
    S3 -->|"mutex etc."| Utils
    HDFS -->|"mutex etc."| Utils
    SFTP -->|"mutex etc."| Utils
    HTTP -->|"mutex etc."| Utils
    STDIO -->|"mutex etc."| Utils

    %% Protocol to External
    FS -->|"OS calls"| OSFS
    S3 -->|"HTTP calls/AWS API"| S3Svc
    HDFS -->|"RPC calls"| HDFSCluster
    SFTP -->|"SSH calls"| SFTPServer
    HTTP -->|"HTTP requests"| HTTPServer
    STDIO -->|"stdin/stdout"| STDIOExt

    %% Click Events
    click CLI "https://github.com/megvii-research/megfile/blob/main/megfile/cli.py"
    click Functional "https://github.com/megvii-research/megfile/blob/main/megfile/smart.py"
    click SmartPath "https://github.com/megvii-research/megfile/blob/main/megfile/smart_path.py"
    click Config "https://github.com/megvii-research/megfile/blob/main/megfile/config.py"
    click Dispatcher "https://github.com/megvii-research/megfile/blob/main/megfile/interfaces.py"
    click FS "https://github.com/megvii-research/megfile/blob/main/megfile/fs.py"
    click FS "https://github.com/megvii-research/megfile/blob/main/megfile/fs_path.py"
    click S3 "https://github.com/megvii-research/megfile/blob/main/megfile/s3.py"
    click S3 "https://github.com/megvii-research/megfile/blob/main/megfile/s3_path.py"
    click HDFS "https://github.com/megvii-research/megfile/blob/main/megfile/hdfs.py"
    click HDFS "https://github.com/megvii-research/megfile/blob/main/megfile/hdfs_path.py"
    click SFTP "https://github.com/megvii-research/megfile/blob/main/megfile/sftp.py"
    click SFTP "https://github.com/megvii-research/megfile/blob/main/megfile/sftp_path.py"
    click HTTP "https://github.com/megvii-research/megfile/blob/main/megfile/http.py"
    click HTTP "https://github.com/megvii-research/megfile/blob/main/megfile/http_path.py"
    click STDIO "https://github.com/megvii-research/megfile/blob/main/megfile/stdio.py"
    click STDIO "https://github.com/megvii-research/megfile/blob/main/megfile/stdio_path.py"
    click Lib "https://github.com/megvii-research/megfile/tree/main/megfile/lib/"
    click Utils "https://github.com/megvii-research/megfile/tree/main/megfile/utils/"
    click ErrorDef "https://github.com/megvii-research/megfile/blob/main/megfile/errors.py"
    click Version "https://github.com/megvii-research/megfile/blob/main/megfile/version.py"
    click Pathlike "https://github.com/megvii-research/megfile/blob/main/megfile/pathlike.py"

    %% Styles
    classDef entry fill:#a2fca0,stroke:#333,stroke-width:1px
    classDef core fill:#a0c4fc,stroke:#333,stroke-width:1px
    classDef protocol fill:#ffb360,stroke:#333,stroke-width:1px
    classDef util fill:#fdffb6,stroke:#333,stroke-width:1px
    classDef external fill:#d3d3d3,stroke:#333,stroke-width:1px
```
