# jeanibarz/probir architecture

> Architecture diagram of the GitHub repository jeanibarz/probir, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jeanibarz/probir
- Repository: https://github.com/jeanibarz/probir

## Overview

##### Project Type and Purpose
- PROBIR is an open-source data-capture and curation tool, not a user-facing web app.
- It consists of two co-ordinated subsystems:
a) A proxy-based traffic interceptor (mitmproxy addon) that logs HTTP/S data into an SQLite database.
b) A modular ETL-style pipeline that reads that database, processes traces in discrete steps, and outputs curated datasets (e.g., for Hugging Face).

##### File Structure
- Top-level scripts and config:
- start_proxy.sh – bootstraps mitmdump with probir.py
- pipeline.yaml – defines ordered pipeline steps and parameters
- .env example – environment variables for database path, target domains, LLM hosts, etc.
- src/:
- probir.py – mitmproxy addon capturing traffic to SQLite
- run_pipeline.py – orchestrator that reads pipeline.yaml and invokes each step
- step*.py – individual processing modules (anonymization, session identification, complexity scoring, pattern analysis)
- push_hf_dataset.py, inspect_db.py – utility CLIs
- common_utils.py – shared logging, I/O, arg-parsing
- tests/: Pytest suite validating each step and utility

##### Readme Insights
- The “Core Components” section already describes the two subsystems.
- Emphasis on checkpointing, resume, HuggingFace integration, and LLM-based anonymization.

- Proxy Interceptor (mitmdump + probir.py)
- SQLite Database (raw traffic store)
- Pipeline Orchestrator (run_pipeline.py)
- Processing Steps (step1–step3 scripts)
- LLM Service (Ollama host)
- Dataset Output (JSONL files / Hugging Face Hub)
- Auxiliary CLIs (inspect_db, push_hf_dataset)

1. Proxy → writes HTTP/S requests & responses → SQLite DB.
2. run_pipeline reads DB → dispatches to each step in pipeline.yaml.
3. Step modules read intermediate files → emit new artifacts or checkpoint → next step.
4. Anonymization steps call external LLM service (Ollama).
5. Final step outputs dataset → push_hf_dataset pushes to Hugging Face Hub.
6. inspect_db can query DB at any time (optional path).

c. Architectural Patterns & Principles:
- Modular, pluggable pipeline driven by configuration (pipeline.yaml).
- Checkpoint & resume for fault tolerance.
- Separation of concerns: capture vs. processing.
- Shared utilities to enforce consistency (common_utils).

d. Key Technologies & Frameworks:
- Python 3.8+
- mitmproxy / mitmdump addon API
- SQLite for lightweight storage
- YAML for pipeline config
- Ollama (LLM host)
- Hugging Face Hub integration
- Pytest for testing

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Proxy Interceptor Subsystem
    subgraph "Proxy Interceptor"
        direction TB
        A1["start_proxy.sh"]:::config
        A2["probir.py"]:::script
    end
    click A1 "https://github.com/jeanibarz/probir/blob/main/start_proxy.sh"
    click A2 "https://github.com/jeanibarz/probir/blob/main/src/probir.py"

    %% Database
    DB["SQLite Database"]:::db

    A1 -->|"boots mitmdump with addon"| A2
    A2 -->|"writes HTTP/S traffic"| DB

    %% Pipeline Configuration
    subgraph "Configuration"
        direction TB
        C1["pipeline.yaml"]:::config
        C2[".env.example"]:::config
    end
    click C1 "https://github.com/jeanibarz/probir/blob/main/pipeline.yaml"
    click C2 "https://github.com/jeanibarz/probir/blob/main/.env.example"

    %% ETL Pipeline Orchestrator
    subgraph "ETL Pipeline"
        direction TB
        O1["run_pipeline.py"]:::script
        O2["common_utils.py"]:::script

        O1 -->|"reads config"| C1
        O1 -->|"loads env vars"| C2
        DB -->|"provides raw data"| O1

        %% Extraction Step
        E1["extract_valid_traces.py"]:::script
        O1 -->|"executes step: extract traces"| E1
        E1 -->|"outputs traces.jsonl"| S1A

        %% Step 1: Anonymization
        subgraph "Step 1: Anonymization"
            direction TB
            S1A["step1_anonymize_data.py"]:::script
            S1B["step1b_anonymize_llm.py"]:::script
            S1C["step1c_session_aware_llm_anonymize.py"]:::script
        end
        click E1 "https://github.com/jeanibarz/probir/blob/main/src/extract_valid_traces.py"
        click S1A "https://github.com/jeanibarz/probir/blob/main/src/step1_anonymize_data.py"
        click S1B "https://github.com/jeanibarz/probir/blob/main/src/step1b_anonymize_llm.py"
        click S1C "https://github.com/jeanibarz/probir/blob/main/src/step1c_session_aware_llm_anonymize.py"

        E1 -->|"feeds raw traces"| S1A
        S1A -->|"anonymized data"| S1B
        S1B -->|"LLM-assisted anon"| S1C
        S1C -->|"anonymized traces.jsonl"| S2

        %% Step 2: Scoring and Sessions
        subgraph "Step 2: Complexity & Sessions"
            direction TB
            S2C["step2_score_complexity.py"]:::script
            S2B["step2b_identify_sessions.py"]:::script
        end
        click S2C "https://github.com/jeanibarz/probir/blob/main/src/step2_score_complexity.py"
        click S2B "https://github.com/jeanibarz/probir/blob/main/src/step2b_identify_sessions.py"

        S1C -->|"data"| S2C
        S2C -->|"scored traces"| S2B
        S2B -->|"session-tagged data"| S3

        %% Step 3: Pattern Analysis
        S3["step3_analyze_correction_patterns.py"]:::script
        click S3 "https://github.com/jeanibarz/probir/blob/main/src/step3_analyze_correction_patterns.py"
        S2B -->|"input data"| S3
        S3 -->|"patterns.jsonl"| O2

        %% Output
        O2 -->|"final dataset.jsonl"| D1["Dataset Output"]:::external
        D1 -->|"push to HF"| HF["Hugging Face Hub"]:::cloud
    end
    click O1 "https://github.com/jeanibarz/probir/blob/main/src/run_pipeline.py"
    click O2 "https://github.com/jeanibarz/probir/blob/main/src/common_utils.py"

    %% External LLM Service
    Oll["Ollama LLM Service"]:::cloud
    S1B -->|"calls LLM API"| Oll
    S1C -->|"calls LLM API"| Oll

    %% Auxiliary CLI Tools
    subgraph "CLI Tools"
        direction TB
        CLI1["inspect_db.py"]:::script
        CLI2["create_hf_dataset.py"]:::script
        CLI3["push_hf_dataset.py"]:::script
    end
    click CLI1 "https://github.com/jeanibarz/probir/blob/main/src/inspect_db.py"
    click CLI2 "https://github.com/jeanibarz/probir/blob/main/src/create_hf_dataset.py"
    click CLI3 "https://github.com/jeanibarz/probir/blob/main/src/push_hf_dataset.py"

    CLI1 -.-> DB
    CLI3 -.-> D1

    %% Tests Suite
    TESTS["tests/"]:::tests
    click TESTS "https://github.com/jeanibarz/probir/tree/main/tests/"

    classDef script fill:#DDEEFF,stroke:#000,stroke-width:1px;
    classDef config fill:#DDFFDD,stroke:#000,stroke-width:1px;
    classDef db fill:#FFEEDD,stroke:#000,stroke-width:1px,stroke-dasharray: 5 5;
    classDef external fill:#FFDDFF,stroke:#000,stroke-width:1px;
    classDef cloud fill:#FFDDAA,stroke:#000,stroke-width:1px;
    classDef tests fill:#EEEEEE,stroke:#000,stroke-width:1px,stroke-dasharray: 2 2;
```
