# arq5x/filo architecture

> Architecture diagram of the GitHub repository arq5x/filo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/arq5x/filo
- Repository: https://github.com/arq5x/filo

## Overview

##### Project type and purpose (short)
- Type: Small suite of UNIX-style command-line utilities (C/C++) for file and stream operations (open‑source bioinformatics utilities).
- Purpose: Provide stream-oriented tools that operate on tab-delimited files/streams (groupBy, shuffle, stats) plus a set of small common libraries used by those tools (gzstream, tabFile, lineFileUtilities, fileType, version).

##### File-structure analysis (key observations)
- Top-level:
- README.rst, LICENSE, Makefile — project metadata and build entry point.
- src/ — all source code lives here.
- Binaries/tools (each has its own directory and Makefile):
- src/groupBy/groupBy.cpp (+ .h)
- src/shuffle/shuffle.cpp
- src/stats/stats.cpp
- Shared libraries / common code (reused by the tools):
- src/common/gzstream/ — gz-compatible stream wrapper (third-party wrapper around zlib).
- src/common/tabFile/ — parsing/handling tab-delimited fields.
- src/common/lineFileUtilities/ — line-level helpers (probably reading lines, tokenizing).
- src/common/fileType/ — detect file type (stdin vs file vs gz).
- src/common/version/ — versioning utilities.
- Build system:
- Per-directory Makefiles and top-level Makefile — implies GNU Make based build producing standalone binaries.
- Tests/artifacts:
- gzstream/test_* files and gzstream.o — test objects or prebuilt object exists.

##### README insights
- Describes intended CLI behaviors and expected I/O (reads files or stdin, writes stdout).
- Behavior is stream-oriented (can pipe input).
- Tools assume particular input characteristics (e.g., groupBy expects sorted groups).
- No network components, no database backends — purely local, file/stream processing.

##### Granularity & details to include (be exhaustive)
- For gzstream specifically, show that it’s an external/third-party submodule and depends on zlib.
- Both compile-time and runtime relationships:
- Compile-time: common modules are compiled and linked into binaries (Makefiles -> binary).
- Runtime: binaries dynamically/actively call common modules (I/O and parsing) and operate on streaming data.
- I/O modes:
- stdin <-> stdout pipes
- File input (plain/gz) with fileType detection and gzstream transparently handling gzip.
- Test artifacts and note where they live (gzstream test files).

##### Final hints / quality checks
- Separate concerns: parsing I/O (common/*) vs algorithmic processing (each tool’s .cpp).
- Keep it faithful to the file tree: show each named common module and map which binaries call them.

Summary (one-line)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Top actors
    User(["User / CLI scripts"]):::actor
    Upstream(["Upstream tools / scripts"]):::actor

    %% Inputs (data stores)
    subgraph "Input Sources" 
        direction TB
        FSPlain(["Plain files\n(flat text)"]):::data
        FSGz(["Gzipped files\n(.gz)"]):::data
        STDIN(["stdin (pipe)"]):::data
    end

    %% Binaries (runtime containers)
    subgraph "CLI Binaries (Runtime)" 
        direction TB
        groupBy["groupBy\nsrc/groupBy/groupBy.cpp"]:::binary
        shuffle["shuffle\nsrc/shuffle/shuffle.cpp"]:::binary
        stats["stats\nsrc/stats/stats.cpp"]:::binary
        moreTools["Other future tools\n(src/... )"]:::binary
    end

    %% Outputs
    subgraph "Outputs / Consumers"
        direction TB
        STDOUT(["stdout (pipe)"]):::data
        OutFile(["Output files"]):::data
        Downstream(["Downstream tools / scripts"]):::actor
    end

    %% Common libraries / modules
    subgraph "Shared Modules (common/)" 
        direction TB
        gzstream["gzstream\nsrc/common/gzstream/gzstream.C"]:::lib
        fileType["fileType\nsrc/common/fileType/fileType.cpp"]:::lib
        lineFile["lineFileUtilities\nsrc/common/lineFileUtilities/lineFileUtilities.h"]:::lib
        tabFile["tabFile\nsrc/common/tabFile/tabFile.cpp"]:::lib
        version["version\nsrc/common/version/version.h"]:::lib
    end

    %% Build system and artifacts
    subgraph "Build & Dev Artifacts"
        direction TB
        TopMake["Top-level Makefile\nMakefile, README.rst, LICENSE"]:::build
        PerMake["Per-directory Makefiles\n(src/*/Makefile)"]:::build
        Tests["gzstream tests & objects\nsrc/common/gzstream/*.o tests"]:::artifact
        Compiler["Compiler (g++, clang)\nGNU Make"]:::dep
    end

    %% External native deps
    subgraph "External Native Dependencies"
        direction TB
        zlib["zlib (native lib)"]:::dep
        cppRt["C++ runtime / stdlib"]:::dep
    end

    %% Legend / Notes
    Notes(["Notes:\n- Stream-oriented: stdin/stdout/pipes\n- groupBy expects sorted input\n- gz transparent via gzstream/zlib"]):::note

    %% Connections: Data flows (top-down)
    User -->|stdin_pipe| STDIN
    Upstream -->|pipe_or_file| STDIN
    FSPlain -->|file_read| groupBy
    FSPlain -->|file_read| shuffle
    FSPlain -->|file_read| stats
    FSGz -->|file_read| fileType
    STDIN -->|stdin_reads| fileType

    %% fileType / gzstream usage runtime
    fileType -->|detects_gz| gzstream
    gzstream -->|provides_istream| groupBy
    gzstream -->|provides_istream| shuffle
    gzstream -->|provides_istream| stats

    %% parsing libs used by binaries
    groupBy -->|uses| tabFile
    groupBy -->|uses| lineFile
    groupBy -->|uses| fileType
    groupBy -->|uses| gzstream
    groupBy -->|uses| version

    shuffle -->|uses| tabFile
    shuffle -->|uses| lineFile
    shuffle -->|uses| fileType
    shuffle -->|uses| gzstream
    shuffle -->|uses| version

    stats -->|uses| tabFile
    stats -->|uses| lineFile
    stats -->|uses| fileType
    stats -->|uses| gzstream
    stats -->|uses| version

    %% Outputs
    groupBy -->|writes_stdout| STDOUT
    shuffle -->|writes_stdout| STDOUT
    stats -->|writes_stdout| STDOUT

    groupBy -->|writes_file| OutFile
    shuffle -->|writes_file| OutFile
    stats -->|writes_file| OutFile

    STDOUT -->|pipe_to| Downstream
    OutFile -->|read_by| Downstream

    %% Build-time relationships
    TopMake -->|orchestrates| PerMake
    PerMake -->|invokes| Compiler
    PerMake -->|compiles_links| groupBy
    PerMake -->|compiles_links| shuffle
    PerMake -->|compiles_links| stats

    %% Common modules compiled/linked into binaries
    tabFile -->|compiled_into| groupBy
    tabFile -->|compiled_into| shuffle
    tabFile -->|compiled_into| stats

    lineFile -->|compiled_into| groupBy
    lineFile -->|compiled_into| shuffle
    lineFile -->|compiled_into| stats

    fileType -->|compiled_into| groupBy
    fileType -->|compiled_into| shuffle
    fileType -->|compiled_into| stats

    gzstream -->|compiled_into| groupBy
    gzstream -->|compiled_into| shuffle
    gzstream -->|compiled_into| stats

    version -->|compiled_into| groupBy
    version -->|compiled_into| shuffle
    version -->|compiled_into| stats

    Tests -->|exercise| gzstream

    %% External deps relationships
    gzstream -->|depends_on| zlib
    groupBy -->|depends_on| cppRt
    shuffle -->|depends_on| cppRt
    stats -->|depends_on| cppRt
    Compiler -->|links_against| cppRt

    %% Layout hints to keep vertical
    classDef actor fill:#f9f,stroke:#333,stroke-width:1px,color:#111;
    classDef data fill:#e6e6e6,stroke:#666,stroke-width:1px,color:#111;
    classDef binary fill:#cde8ff,stroke:#0366d6,stroke-width:1px,color:#042a4b;
    classDef lib fill:#d8f3d8,stroke:#2a8a2a,stroke-width:1px,color:#08360a;
    classDef build fill:#efe6ff,stroke:#7a4fc2,stroke-width:1px,color:#2e0740;
    classDef dep fill:#ffd9b3,stroke:#d97706,stroke-width:1px,color:#4b2b00;
    classDef artifact fill:#f0f0f0,stroke:#999,stroke-width:1px,color:#333;
    classDef note fill:#fff3cd,stroke:#b8860b,stroke-width:1px,color:#4b2b00;

    %% Click events for mapped components (paths only)
    click groupBy "https://github.com/arq5x/filo/blob/master/src/groupBy/groupBy.cpp"
    click groupBy "https://github.com/arq5x/filo/blob/master/src/groupBy/groupBy.h"
    click groupBy "https://github.com/arq5x/filo/tree/master/src/groupBy/Makefile"
    click shuffle "https://github.com/arq5x/filo/blob/master/src/shuffle/shuffle.cpp"
    click shuffle "https://github.com/arq5x/filo/tree/master/src/shuffle/Makefile"
    click stats "https://github.com/arq5x/filo/blob/master/src/stats/stats.cpp"
    click stats "https://github.com/arq5x/filo/tree/master/src/stats/Makefile"

    click gzstream "https://github.com/arq5x/filo/blob/master/src/common/gzstream/gzstream.C"
    click gzstream "https://github.com/arq5x/filo/blob/master/src/common/gzstream/gzstream.h"
    click gzstream "https://github.com/arq5x/filo/tree/master/src/common/gzstream/Makefile"
    click gzstream "https://github.com/arq5x/filo/blob/master/src/common/gzstream/COPYING.LIB"
    click gzstream "https://github.com/arq5x/filo/tree/master/src/common/gzstream/README"
    click gzstream "https://github.com/arq5x/filo/blob/master/src/common/gzstream/gzstream.o"
    click gzstream "https://github.com/arq5x/filo/blob/master/src/common/gzstream/test_gunzip.o"
    click gzstream "https://github.com/arq5x/filo/blob/master/src/common/gzstream/test_gzip.o"

    click tabFile "https://github.com/arq5x/filo/blob/master/src/common/tabFile/tabFile.cpp"
    click tabFile "https://github.com/arq5x/filo/blob/master/src/common/tabFile/tabFile.h"
    click tabFile "https://github.com/arq5x/filo/tree/master/src/common/tabFile/Makefile"

    click lineFile "https://github.com/arq5x/filo/blob/master/src/common/lineFileUtilities/lineFileUtilities.h"

    click fileType "https://github.com/arq5x/filo/blob/master/src/common/fileType/fileType.cpp"
    click fileType "https://github.com/arq5x/filo/blob/master/src/common/fileType/fileType.h"
    click fileType "https://github.com/arq5x/filo/tree/master/src/common/fileType/Makefile"

    click version "https://github.com/arq5x/filo/blob/master/src/common/version/version.h"

    click TopMake "https://github.com/arq5x/filo/tree/master/Makefile"
    click TopMake "https://github.com/arq5x/filo/blob/master/README.rst"
    click TopMake "https://github.com/arq5x/filo/tree/master/LICENSE"

    click PerMake "https://github.com/arq5x/filo/tree/master/src/groupBy/Makefile"
    click PerMake "https://github.com/arq5x/filo/tree/master/src/shuffle/Makefile"
    click PerMake "https://github.com/arq5x/filo/tree/master/src/stats/Makefile"
    click PerMake "https://github.com/arq5x/filo/tree/master/src/common/tabFile/Makefile"
    click PerMake "https://github.com/arq5x/filo/tree/master/src/common/gzstream/Makefile"
    click PerMake "https://github.com/arq5x/filo/tree/master/src/common/fileType/Makefile"

    %% Small annotations for important constraints
    note_sorted["groupByexpects_sorted_input"]:::note
    note_stream["Stream-friendly:use_pipes"]:::note
    note_gz["Gz_support_via_gzstream+zlib"]:::note

    note_sorted --> groupBy
    note_stream --> STDIN
    note_gz --> gzstream

    %% Force vertical flow by linking top to bottom
    User -->|invokes| groupBy
    Upstream -->|pipes_to| groupBy
    User -->|invokes| shuffle
    User -->|invokes| stats
```
