How mateusfernandes25/sectrum works
Project Type and Purpose
• This is a machine-learning/NLP research pipeline rather than a classic “frontend/backend” app.
• Purpose: fine-tune a LLaMA-7B model on a custom Portuguese information-security Q&A database using QLoRA, evaluate its performance, and store artifacts & results.
File Structure
Top-level:
• Notebooks:
• Creating Data Base.ipynb – builds/prepares the .xlsx dataset
• Training.ipynb – runs the QLoRA fine-tuning (Hugging Face Transformers + Accelerate)
• Evaluation.ipynb – measures model performance (semantic similarity, entailment, QA)
• Security Data Base Portuguese.xlsx – raw Q&A dataset
• model/ – contains fine-tuned LLaMA artifacts (config.json, adapter_model.bin, tokenizer, etc.)
• results/ – stores evaluation outputs: charts, spreadsheets, evidence
• README.md – describes methodology, stack, goals
README
• Confirms core components: data collection, QLoRA methodology, fine-tuning, evaluation.
• Technology stack: Python ≥3.8, PyTorch, Hugging Face Transformers, Tokenizers, Accelerate.
Main Components & Their Interactions
a. Components
1. Data Source
• Security Data Base Portuguese.xlsx (Q&A pairs)
2. Preprocessing
• “Creating Data Base.ipynb”: cleans, formats, tokenizes inputs
3. Training Pipeline
• “Training.ipynb”: invokes QLoRA adapter_config, uses PyTorch + Accelerate to fine-tune LLaMA
4. Model Artifacts
• model/ directory (adapter_model.bin, config.json, tokenizer files)
5. Evaluation
• “Evaluation.ipynb”: loads fine-tuned model, computes metrics, produces charts
6. Results Storage
• results/ directory with charts (.xlsx), evidence files
7. External Dependencies
• Hugging Face Hub (optional for checkpoints), Cloud/Local GPU cluster, Python libraries
b. Relationships & Data Flow
• Tokenized dataset → Training notebook → Fine-tuned model artifacts
• Model artifacts → Evaluation notebook → Metrics & charts
• Metrics & charts → results/ directory
c. Architectural Patterns & Design Principles
• Pipeline pattern: linear stages (Ingest → Preprocess → Train → Evaluate → Store)
• Separation of concerns: each Jupyter notebook handles one stage
• Artifact versioning: model/ directory holds versioned adapter + tokenizer configs
d. Key Technologies
• QLoRA adapter framework (quantized low-rank adaptation)
• Hugging Face Transformers & Tokenizer libraries
• PyTorch + Accelerate for distributed GPU training
• Jupyter Notebooks for experimentation
• Off to the side, show “Hugging Face Hub” cloud connected to Training and Evaluation (for model checkpoints).