# matz/streem architecture

> Architecture diagram of the GitHub repository matz/streem, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/matz/streem
- Repository: https://github.com/matz/streem

## Overview

##### Project type and purpose
- Streem is a standalone, stream-based concurrent scripting language runtime and interpreter written in C.
- It compiles a small DSL (inspired by shell pipelines, Ruby, Erlang) into an in-memory dataflow graph and executes it via an event loop and worker threads.

##### File structure
- Top-level: Dockerfile, Makefile, CI workflows, docs/, examples/
- src/:
- lex.l / parse.y – scanner and parser (Flex/Bison)
- main.c – command-line entry point
- init.c, core.c, graph.c, node.c – AST → dataflow graph construction
- exec.c, io.c, pollfd.c, signal.c – event loop & OS integration (Reactor pattern)
- ncpu.c, latch.c, atomic.h – concurrency primitives & worker threads
- math.c, array.c, csv.c, kvs.c, sort.c, random.c, etc. – built-in stream operators & libraries
- utils/ & vagrant/ for dev environment scripts

##### README for additional insights
- Describes the pipeline DSL (`stdin | stdout`, `seq(100) | map{…} | stdout`)
- Key concepts: streams, nodes (operators), event loop, mapping/filtering functions, concurrency

##### Detail without over-splitting
- Every major phase, module group, and external dependency
- Keep component granularity at the level of “parser vs. execution engine” rather than every .c file

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Layers top to bottom
    %% Frontend Layer
    subgraph "Frontend Layer"
        direction TB
        CLI["CLI Frontend"]:::frontend
        Parser["Scanner & Parser<br/>(Flex → Bison)"]:::frontend
    end

    %% IR Layer
    subgraph "Intermediate Representation" 
        direction TB
        AST(("AST / Semantic Analyzer")):::ir
        Graph(("Dataflow DAG")):::ir
    end

    %% Runtime Layer
    subgraph "Execution Engine" 
        direction TB
        subgraph "Event Loop & OS Integration" 
            direction TB
            EventLoop["Event Loop & Reactor"]:::runtime
        end
        subgraph "Scheduler / Worker Threads"
            direction TB
            Scheduler["Worker Pool & Primitives"]:::runtime
        end
    end

    %% Modules Layer
    subgraph "Operator Modules" 
        direction TB
        IOOps["I/O Operators"]:::modules
        DataStruct["Data Structures"]:::modules
        MathOps["Math & Randomness"]:::modules
        MiscOps["Misc Built-ins"]:::modules
    end

    %% External OS Interfaces
    OS["OS Interfaces<br/>(FDs, Sockets, Subprocesses)"]:::external

    %% Examples and Docs
    Examples["User Scripts / Examples"]:::external
    Docs["Built-in Library Docs"]:::external

    %% Connections
    CLI -->|"invokes"| Parser
    Parser -->|"AST"| AST
    AST -->|"AST"| Graph
    Graph -->|"DAG of Nodes"| EventLoop
    EventLoop -->|"registers I/O"| OS
    EventLoop -->|"dispatches data tokens"| Scheduler
    Scheduler -->|"invokes handlers"| IOOps
    Scheduler --> IOOps
    Scheduler --> DataStruct
    Scheduler --> MathOps
    Scheduler --> MiscOps
    IOOps --> OS
    Examples --> CLI
    Docs --> MathOps
    Docs --> IOOps
    Docs --> DataStruct
    Docs --> MiscOps

    %% Click Events
    click CLI "https://github.com/matz/streem/blob/master/src/main.c"
    click CLI "https://github.com/matz/streem/blob/master/src/args.c"
    click Parser "https://github.com/matz/streem/blob/master/src/lex.l"
    click Parser "https://github.com/matz/streem/blob/master/src/parse.y"
    click AST "https://github.com/matz/streem/blob/master/src/core.c"
    click AST "https://github.com/matz/streem/blob/master/src/node.c"
    click AST "https://github.com/matz/streem/blob/master/src/node.h"
    click AST "https://github.com/matz/streem/blob/master/src/init.c"
    click Graph "https://github.com/matz/streem/blob/master/src/graph.c"
    click Graph "https://github.com/matz/streem/blob/master/src/node.c"
    click Graph "https://github.com/matz/streem/blob/master/src/node.h"
    click EventLoop "https://github.com/matz/streem/blob/master/src/exec.c"
    click EventLoop "https://github.com/matz/streem/blob/master/src/pollfd.c"
    click EventLoop "https://github.com/matz/streem/blob/master/src/pollfd.h"
    click EventLoop "https://github.com/matz/streem/blob/master/src/signal.c"
    click EventLoop "https://github.com/matz/streem/blob/master/src/io.c"
    click Scheduler "https://github.com/matz/streem/blob/master/src/ncpu.c"
    click Scheduler "https://github.com/matz/streem/blob/master/src/latch.c"
    click Scheduler "https://github.com/matz/streem/blob/master/src/atomic.h"
    click IOOps "https://github.com/matz/streem/blob/master/src/io.c"
    click IOOps "https://github.com/matz/streem/blob/master/src/csv.c"
    click IOOps "https://github.com/matz/streem/blob/master/src/socket.c"
    click IOOps "https://github.com/matz/streem/blob/master/src/exec.c"
    click DataStruct "https://github.com/matz/streem/blob/master/src/array.c"
    click DataStruct "https://github.com/matz/streem/blob/master/src/kvs.c"
    click DataStruct "https://github.com/matz/streem/blob/master/src/queue.c"
    click DataStruct "https://github.com/matz/streem/blob/master/src/queue.h"
    click MathOps "https://github.com/matz/streem/blob/master/src/math.c"
    click MathOps "https://github.com/matz/streem/blob/master/src/random.c"
    click MathOps "https://github.com/matz/streem/blob/master/src/sort.c"
    click MiscOps "https://github.com/matz/streem/blob/master/src/iter.c"
    click MiscOps "https://github.com/matz/streem/blob/master/src/ns.c"
    click MiscOps "https://github.com/matz/streem/blob/master/src/number.c"
    click MiscOps "https://github.com/matz/streem/blob/master/src/stat.c"
    click MiscOps "https://github.com/matz/streem/blob/master/src/string.c"
    click MiscOps "https://github.com/matz/streem/blob/master/src/time.c"
    click MiscOps "https://github.com/matz/streem/blob/master/src/env.c"
    click OS "https://github.com/matz/streem/blob/master/src/io.c"
    click OS "https://github.com/matz/streem/blob/master/src/exec.c"
    click OS "https://github.com/matz/streem/blob/master/src/socket.c"
    click OS "https://github.com/matz/streem/blob/master/src/pollfd.c"
    click OS "https://github.com/matz/streem/blob/master/src/signal.c"
    click Examples "https://github.com/matz/streem/tree/master/examples/"
    click Docs "https://github.com/matz/streem/blob/master/doc/library/math.md"

    %% Styles
    classDef frontend fill:#FFEB3B,stroke:#FBC02D,color:#000
    classDef ir fill:#FFCC80,stroke:#FB8C00,color:#000
    classDef runtime fill:#A5D6A7,stroke:#388E3C,color:#000
    classDef modules fill:#90CAF9,stroke:#1976D2,color:#000
    classDef external fill:#CFD8DC,stroke:#607D8B,color:#000
```
