How jeanibarz/probir works
Project Type and Purpose
• PROBIR is an open-source data-capture and curation tool, not a user-facing web app.
• It consists of two co-ordinated subsystems:
a) A proxy-based traffic interceptor (mitmproxy addon) that logs HTTP/S data into an SQLite database.
b) A modular ETL-style pipeline that reads that database, processes traces in discrete steps, and outputs curated datasets (e.g., for Hugging Face).
File Structure
• Top-level scripts and config:
• start_proxy.sh – bootstraps mitmdump with probir.py
• pipeline.yaml – defines ordered pipeline steps and parameters
• .env example – environment variables for database path, target domains, LLM hosts, etc.
• src/:
• probir.py – mitmproxy addon capturing traffic to SQLite
• run_pipeline.py – orchestrator that reads pipeline.yaml and invokes each step
• step*.py – individual processing modules (anonymization, session identification, complexity scoring, pattern analysis)
• push_hf_dataset.py, inspect_db.py – utility CLIs
• common_utils.py – shared logging, I/O, arg-parsing
• tests/: Pytest suite validating each step and utility
Readme Insights
• The “Core Components” section already describes the two subsystems.
• Emphasis on checkpointing, resume, HuggingFace integration, and LLM-based anonymization.
• Proxy Interceptor (mitmdump + probir.py)
• SQLite Database (raw traffic store)
• Pipeline Orchestrator (run_pipeline.py)
• Processing Steps (step1–step3 scripts)
• LLM Service (Ollama host)
• Dataset Output (JSONL files / Hugging Face Hub)
• Auxiliary CLIs (inspect_db, push_hf_dataset)
1. Proxy → writes HTTP/S requests & responses → SQLite DB.
2. run_pipeline reads DB → dispatches to each step in pipeline.yaml.
3. Step modules read intermediate files → emit new artifacts or checkpoint → next step.
4. Anonymization steps call external LLM service (Ollama).
5. Final step outputs dataset → push_hf_dataset pushes to Hugging Face Hub.
6. inspect_db can query DB at any time (optional path).
c. Architectural Patterns & Principles:
• Modular, pluggable pipeline driven by configuration (pipeline.yaml).
• Checkpoint & resume for fault tolerance.
• Separation of concerns: capture vs. processing.
• Shared utilities to enforce consistency (common_utils).
d. Key Technologies & Frameworks:
• Python 3.8+
• mitmproxy / mitmdump addon API
• SQLite for lightweight storage
• YAML for pipeline config
• Ollama (LLM host)
• Hugging Face Hub integration
• Pytest for testing