# mateusfernandes25/sectrum architecture

> Architecture diagram of the GitHub repository mateusfernandes25/sectrum, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mateusfernandes25/sectrum
- Repository: https://github.com/mateusfernandes25/sectrum

## Overview

##### Project Type and Purpose
- This is a machine-learning/NLP research pipeline rather than a classic “frontend/backend” app.
- Purpose: fine-tune a LLaMA-7B model on a custom Portuguese information-security Q&A database using QLoRA, evaluate its performance, and store artifacts & results.

##### File Structure
Top-level:
- Notebooks:
- Creating Data Base.ipynb – builds/prepares the .xlsx dataset
- Training.ipynb – runs the QLoRA fine-tuning (Hugging Face Transformers + Accelerate)
- Evaluation.ipynb – measures model performance (semantic similarity, entailment, QA)
- Security Data Base Portuguese.xlsx – raw Q&A dataset
- model/ – contains fine-tuned LLaMA artifacts (config.json, adapter_model.bin, tokenizer, etc.)
- results/ – stores evaluation outputs: charts, spreadsheets, evidence
- README.md – describes methodology, stack, goals

##### README
- Confirms core components: data collection, QLoRA methodology, fine-tuning, evaluation.
- Technology stack: Python ≥3.8, PyTorch, Hugging Face Transformers, Tokenizers, Accelerate.

##### Main Components & Their Interactions
a. Components
1. Data Source
- Security Data Base Portuguese.xlsx (Q&A pairs)
2. Preprocessing
- “Creating Data Base.ipynb”: cleans, formats, tokenizes inputs
3. Training Pipeline
- “Training.ipynb”: invokes QLoRA adapter_config, uses PyTorch + Accelerate to fine-tune LLaMA
4. Model Artifacts
- model/ directory (adapter_model.bin, config.json, tokenizer files)
5. Evaluation
- “Evaluation.ipynb”: loads fine-tuned model, computes metrics, produces charts
6. Results Storage
- results/ directory with charts (.xlsx), evidence files
7. External Dependencies
- Hugging Face Hub (optional for checkpoints), Cloud/Local GPU cluster, Python libraries

b. Relationships & Data Flow
- Tokenized dataset → Training notebook → Fine-tuned model artifacts
- Model artifacts → Evaluation notebook → Metrics & charts
- Metrics & charts → results/ directory

c. Architectural Patterns & Design Principles
- Pipeline pattern: linear stages (Ingest → Preprocess → Train → Evaluate → Store)
- Separation of concerns: each Jupyter notebook handles one stage
- Artifact versioning: model/ directory holds versioned adapter + tokenizer configs

d. Key Technologies
- QLoRA adapter framework (quantized low-rank adaptation)
- Hugging Face Transformers & Tokenizer libraries
- PyTorch + Accelerate for distributed GPU training
- Jupyter Notebooks for experimentation

- Off to the side, show “Hugging Face Hub” cloud connected to Training and Evaluation (for model checkpoints).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Components
    DataSource["Security Data Base Portuguese.xlsx"]:::data
    Tokenized["Tokenized Dataset\n(Intermediary)"]:::data
    Results["results/ (charts/, evidence/)"]:::artifact

    %% Compute Stages
    Preprocessing["Creating Data Base.ipynb\n(Preprocessing)"]:::compute
    Training["Training.ipynb\n(QLoRA + LLaMA fine-tuning)"]:::compute
    Evaluation["Evaluation.ipynb\n(Metrics, Charts)"]:::compute

    %% Artifact Storage
    ModelDir["model/ (adapter_model.bin,\nconfig.json, tokenizer files)"]:::artifact

    %% External Dependencies
    Hub["Hugging Face Hub"]:::external
    GPU["GPU Cluster"]:::external

    %% Documentation
    subgraph "Documentation"
        DocProject["README.md"]:::artifact
        DocModel["model/README.md"]:::artifact
    end

    %% Pipeline Flow
    DataSource -->|raw dataset| Preprocessing
    Preprocessing -->|tokenized data| Tokenized
    Tokenized -->|training input| Training
    Training -->|fine-tuned artifacts| ModelDir
    ModelDir -->|load model| Evaluation
    Evaluation -->|metrics & charts| Results

    %% External Links
    Hub -.->|checkpoints| Training
    Hub -.->|optional downloads| Evaluation
    GPU -.->|compute resources| Preprocessing
    GPU -.->|compute resources| Training
    GPU -.->|compute resources| Evaluation

    %% Click Events
    click DataSource "https://github.com/mateusfernandes25/sectrum/blob/main/Security Data Base Portuguese.xlsx"
    click Preprocessing "https://github.com/mateusfernandes25/sectrum/blob/main/Creating Data Base.ipynb"
    click Training "https://github.com/mateusfernandes25/sectrum/blob/main/Training.ipynb"
    click ModelDir "https://github.com/mateusfernandes25/sectrum/tree/main/model/"
    click Evaluation "https://github.com/mateusfernandes25/sectrum/blob/main/Evaluation.ipynb"
    click Results "https://github.com/mateusfernandes25/sectrum/tree/main/results/"
    click DocProject "https://github.com/mateusfernandes25/sectrum/blob/main/README.md"
    click DocModel "https://github.com/mateusfernandes25/sectrum/blob/main/model/README.md"

    %% Styles
    classDef data fill:#D6EFFF,stroke:#0366D6,color:#0366D6;
    classDef compute fill:#D2F8D2,stroke:#06A406,color:#06A406;
    classDef artifact fill:#FFE5CC,stroke:#E68A00,color:#E68A00;
    classDef external fill:#DDDDDD,stroke:#888888,color:#555555;
```
