# composewell/streamly architecture

> Architecture diagram of the GitHub repository composewell/streamly, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/composewell/streamly
- Repository: https://github.com/composewell/streamly

## Overview

##### Project Type & Overall Purpose
- This isn’t a typical full-stack web app but rather a core library (with supporting utilities) meant to deliver efficient streaming operations.
- Its purpose is to expose a public, high-performance API for streaming data and associated operations (e.g., folds, arrays, parsers) while hiding the complex internal implementation details.
- The project also integrates performance benchmarks, a comprehensive test suite, and CI/CD configurations to ensure reliability and continuous improvement.

##### Main Components / Layers
a. Core Library & Internal Modules:
– “core”: Contains the core engine for Streamly. This folder includes everything from internal data types, control constructs, file system interfaces, network operations, concurrency, and low-level data manipulations.
– “src”: Contains the public API modules for Streamly, such as streaming primitives, folds, arrays, and basic I/O constructs.
b. Benchmarking Infrastructure:
– “benchmark” and “bench-test-lib”: These directories hold benchmarking projects, microbenchmarks, and utilities for performance comparison. They help in testing different stream variants and evaluating optimizations.
c. Testing Infrastructure:
– “test”: This directory contains test cases covering various components (arrays, folds, streams, file system, networking, Unicode etc.) and helps ensure that each part of the system behaves as expected.
d. Build, Configuration, and Targets:
– Top-level build scripts and configuration files (like “cabal.project”, “Setup.hs”, “configure.ac”, “default.nix”, and “targets”) indicate that Cabal, Nix, and other build systems/tools are used to configure and build the project.
e. Documentation & Examples:
– “docs” and “examples”: These directories provide user and developer documentation, guidelines, tutorials, and example usage maintaining the extensibility and developer engagement.
f. CI/CD and Auxiliary Scripts:
– Directories “.github”, “.circleci”, and other related configuration files (e.g., appveyor.yml) show that automated testing and CI workflows are part of the development cycle.

##### Relationships and Interactions
- The Public API in “src” interfaces with the deeper internal implementations found in “core”.
- Higher-level user functionality is built on top of the core streaming and fold abstractions.
- Benchmark and test suites are attached to these libraries to run performance regression and correctness checks.
- The build and configuration layer (cabal files, nix scripts, CI scripts) ties everything together and ensures that changes in the internal components or public API can be verified via automated tests and performance benchmarks.

##### Architectural Patterns & Technologies
- The project follows a modular structure in Haskell, separating concerns with a clear divide between internal (core) and external (public API) modules.
- It employs a functional programming design using Haskell idioms and patterns. This includes composable stream processing, immutability, and high-order functions.
- Frameworks and tools include Cabal for Haskell package management, Nix for reproducible builds, and CI services (GitHub Actions, CircleCI, AppVeyor) to enforce code quality.
- A layered approach is adopted – the core engine’s lower-level modules support the higher-level interfaces provided to end-users.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Build & Configuration"
        BC("Build & Configuration"):::build
    end

    subgraph "Core Library"
        API("Public API Modules"):::public
        CORE("Core Engine/Internal Modules"):::core
    end

    subgraph "Support Infrastructure"
        BENCH("Benchmarking Infrastructure"):::bench
        TEST("Testing Suite"):::test
    end

    subgraph "Documentation & CI/CD"
        DOCS("Documentation & Examples"):::docs
        CICD("CI/CD & Auxiliary Scripts"):::ci
    end

    API -->|"calls"| CORE
    BENCH -->|"benchmarks"| API
    BENCH -->|"benchmarks"| CORE
    TEST -->|"tests"| API
    TEST -->|"tests"| CORE
    BC -->|"builds"| API
    BC -->|"builds"| CORE
    BC -->|"builds"| BENCH
    BC -->|"builds"| TEST
    CICD -->|"orchestrates"| BC
    DOCS -->|"documents"| API

    click CORE "https://github.com/composewell/streamly/tree/master/core"
    click API "https://github.com/composewell/streamly/tree/master/src"
    click BENCH "https://github.com/composewell/streamly/tree/master/benchmark"
    click BENCH "https://github.com/composewell/streamly/tree/master/bench-test-lib"
    click TEST "https://github.com/composewell/streamly/tree/master/test"
    click BC "https://github.com/composewell/streamly/blob/master/cabal.project"
    click BC "https://github.com/composewell/streamly/blob/master/Setup.hs"
    click BC "https://github.com/composewell/streamly/tree/master/targets"
    click BC "https://github.com/composewell/streamly/blob/master/default.nix"
    click DOCS "https://github.com/composewell/streamly/tree/master/docs"
    click DOCS "https://github.com/composewell/streamly/tree/master/examples"
    click CICD "https://github.com/composewell/streamly/blob/master/.github"
    click CICD "https://github.com/composewell/streamly/blob/master/.circleci"
    click CICD "https://github.com/composewell/streamly/blob/master/appveyor.yml"

    classDef core fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef public fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef bench fill:#FFD700,stroke:#000,stroke-width:2px;
    classDef test fill:#FF6347,stroke:#000,stroke-width:2px;
    classDef build fill:#D3D3D3,stroke:#000,stroke-width:2px;
    classDef docs fill:#FFB6C1,stroke:#000,stroke-width:2px;
    classDef ci fill:#9370DB,stroke:#000,stroke-width:2px;
```
