How abhishekn1947/ghibli-diffuser-open-source works
Project type and purpose
• This is an open-source AI tool (“GhibliFilterTool”) that fine-tunes Stable Diffusion v1.5 with LoRA to apply a Studio Ghibli–style filter to images. It’s not a web or mobile app, but a notebook/script-driven ML pipeline intended to run in Google Colab (or any GPU-enabled Python environment).
File structure
• Top level:
• Ghibli-Fine-Tuner.py ‑ a Python script that encapsulates data loading, model initialization, training, and saving
• ghibly_diffuser_v0_2.ipynb ‑ a Colab notebook version of the same pipeline, with cells for environment setup, training, and inference
• README.md, LICENSE
• No frontend/backend split; instead, the “architecture” is a data→training→inference pipeline.
Key insights from the README
• Data lives on Google Drive (Ghibli images + regularization images).
• Dependencies: PyTorch, Hugging Face Diffusers, PEFT, python-dotenv, Pillow.
• Training uses an NVIDIA A100 GPU in Colab Pro; inference occurs in the same environment.
• Base model loaded from Hugging Face; fine-tuned LoRA weights saved locally or back to Drive.
Main components and their relationships
a. Data Storage & Sources
• Google Drive folders for “ghibli_dataset” and “regularization_dataset”
• Optional: Base model artifacts fetched from Hugging Face Model Hub
b. Compute Environment
• Google Colab (A100/T4/V100) or local GPU
• Python runtime loading Ghibli-Fine-Tuner.py / notebook
c. Dependency Layer
• PyTorch
• Diffusers (StableDiffusionPipeline)
• PEFT (LoRA adapter)
• python-dotenv (for config like paths, hyperparameters)
d. Training Pipeline
• DataLoader: reads images from Drive → transforms → batches
• Model init: load Stable Diffusion v1.5 → wrap with LoRA modules
• Optimizer & scheduler: configured per README hyperparameters
• Training loop: forward/backward → update LoRA weights → early stopping
• Checkpointing: save LoRA weights to local disk or Drive
e. Inference Pipeline
• Load base model + fine-tuned LoRA weights
• Accept user image input (file uploader in notebook)
• StableDiffusionPipeline.inference(…) → stylized output image
• Save/display result, optionally push back to Drive
f. External Services
• Hugging Face Model Hub (fetch base model)
• Google Drive (store datasets, load/save checkpoints and outputs)
Architectural patterns and design principles
• Pipeline pattern: clear stages of data ingestion, model training, and inference
• Separation of concerns: data vs. compute vs. model vs. external I/O
• Extensibility points:
• Hyperparameter tuning (LoRA r, alpha, LR, batch size)
• Dataset swap (drop-in new image collections)
• Model variant (change base SD version)
1. Data Sources
2. Compute Environment
3. Dependency Layer
4. Training Pipeline
5. Inference Pipeline
6. External Services
• From Data Sources → Training Pipeline/DataLoader
• From Dependency Layer → Training & Inference
• From Training Pipeline → Checkpoint storage (Drive)
• From Inference Pipeline → Output images → Drive / User
• Relationships:
• The Training and Inference pipelines both depend on the base model from Hugging Face and the PEFT LoRA modules.
• Both pipelines execute within the Google Colab GPU environment.