How dcy0577/bim-command-recommendation works
Project type and purpose
• This is a full-stack predictive modeling system for BIM command recommendation. It spans data engineering (large-scale log processing), ML model development (training transformer-based recommenders), a custom modeling library (transformers4rec + peft), deployment (Triton server, local inference, ensemble scripts), and a real-time web prototype (FastAPI backend + Vue.js frontend).
File structure
• Top-level directories:
• data_processing – pipelines to filter and enrich raw BIM logs
• model – training, evaluation, and utility scripts built on your custom libraries
• deployment – scripts/templates to package, ensemble, and serve models (Triton, local)
• prototype – real-time service that watches logs and serves command suggestions via FastAPI + Vue.js
• transformers4rec & peft – the custom versions of Nvidia’s and Huggingface’s libraries you extend
• visualization – scripts to produce figures for analysis
• Config and orchestration: docker-compose.dev.deploy.yml, conda env yml, Triton config pbtxt
• MVC and micro-services elements: data pipelines are standalone scripts/notebooks; model code lives in its own folder; deployment is service-oriented; prototype is a two-tier (API + front end) app.
Extract additional insights from the README
• Data scale: 32 billion rows of global log data
• Model: transformer backbone + custom feature fusion + dedicated loss + learning strategy
• Deployment: Triton inference server, Python backend, ensemble of PEFT LLMs, local inference for testing
• Prototype: polls Vectorworks logs, returns “next command” suggestions in real time
a. Main components
1. Raw Data Ingestion: BIM logs (flat files / parquet)
2. Data Processing Pipelines: modules in data_processing that filter, group, cluster, augment, and remove redundancy
3. Feature Store / Preprocessed Data: artifacts written to disk or table format for training
4. Model Training & Evaluation: scripts in model/ that load data, build transformer-based model via transformers4rec + peft, run training, evaluate metrics
5. Model Repository: saved checkpoints (model.pt) and Triton model directory built with deploy_template
6. Deployment Services:
• Triton Inference Server (with Python backend & config.pbtxt)
• Local inference script for dev/testing
• Ensemble PEFT LLM script that wraps multiple model variants
7. Real-Time API Service: FastAPI endpoints (in prototype/web_application/fastapi)
8. Frontend Application: Vue.js SPA (in prototype/web_application/frontend)
9. Visualization Tools: standalone scripts to generate plots (not part of runtime)
b. Relationships and interactions
• Data flows from Raw Data → Data Processing → Preprocessed Data
• Preprocessed Data → Model Training → Checkpoints → Model Repository
• Model Repository → Deployment Scripts → Triton Server & Local Inference
• Triton & Local Inference Services → API Service
• API Service ↔ Frontend (user requests recommendation, receives next-command suggestions)
• Visualization reads Preprocessed Data or Model Outputs for offline analysis
c. Architectural patterns & design principles
• Modular pipeline: separate data processing notebooks/scripts by logical stages
• Library reuse: custom extensions (transformers4rec, peft) abstract low-level modeling details
• Service-oriented inference: Triton server for high throughput, local script for dev
• Microservice for real-time recommendations: decoupled FastAPI + SPA
• Configuration as code: conda yml, docker-compose, Triton config
d. Key technologies
• Python, Jupyter notebooks for data pipelines
• PyTorch (through transformers4rec), Huggingface PEFT for model training
• NVIDIA Triton Inference Server (Python backend)
• FastAPI for REST API
• Vue.js + Vite for frontend SPA
• Docker Compose for orchestration
• Conda for environment management
Checklist for the principal software engineer
• Clear, descriptive labels on every component
• External dependencies (e.g., Vectorworks logs, CUDA-enabled GPU)
• Essential subsystems: data_processing, model training, custom library, deployment, API, frontend
• If space allows, insert version or technology callouts (e.g., “Triton v2.x,” “Vue 3 / Vite”)