How data-wrangling-and-visualisation/renai works
Project Type and Purpose
• R.E.N.A.I is a full-stack, data-driven web application for AI-based art exploration and analysis.
• Frontend: React + Material UI for interactive visualizations.
• Backend: Flask API orchestrating ML models, caching results, serving data.
• Auxiliary pipelines: data collection (JSON/image ingestion), pre-processing, embedding and map generation.
File-Structure Analysis
• Top-level:
• react-renai/ → React frontend (Vite config, server.js for production, components, utils).
• server/ → Flask backend (api.py, processing modules, data cache under server/data).
• processing-data/ → standalone scripts generating embeddings and attention maps.
• сollection-data/ → data-ingestion scripts fetching MET JSON batches, saving images, uploading to Mongo.
• docker-compose.yml, Dockerfiles → container setup.
• Patterns: clear client/server separation, offline pipelines for data collection and processing, file-based caching (.npy, .json).
Key README Insights
• Machine-learning modules: EfficientNetB0/DINOv2 (embeddings), ResNet50 (Grad-CAM), ViT (attention).
• Lazy loading & virtualization on frontend.
• REST API endpoints under /api/* for metadata, analysis, visualizations.
• External dependency: Metropolitan Museum of Art Collection API.
Main Components
a. Frontend Layer (React-RenAI)
• Browser client (JS/CSS)
• React components: EmbeddingProjection, ArtSimilarityGraph, AttentionVisualizer, StyleAnalysis, Sidebar, etc.
• Client-side state & session storage
• Dev server (Vite) or production static server (server.js)
b. Backend Layer (Flask API)
• api.py routes
• Processing modules: embeddings.py, Gradcam.py, hitmaps.py, improved_analysis.py, preprocess_artworks.py
• Utilities (utils.py, color_names_ru.py)
• Data cache (server/data/embeddings, server/data/images)
c. Data Ingestion & Pre-Processing Pipelines
• collection-data scripts: JSON fetcher, image_saver, upload_to_mongo
• processing-data scripts producing .npy heatmaps and embedding arrays
d. External Services & Storage
• Metropolitan Museum of Art API (metadata + raw images)
• (Optional) MongoDB for raw JSON storage
• Docker containers for frontend/backend
Interactions & Data Flow
• Data-Ingestion Pipeline:
MET API → collection-data/ → raw JSON, images → (optional) MongoDB
• Pre-Processing Pipeline:
raw images → processing-data/ → .npy embeddings & maps → server/data
• Runtime Backend:
Client ↔ Flask API endpoints → on-demand processing or cached lookup → returns JSON, images, .npy data
• Frontend:
React UI → REST calls → renders visualizations (charts, force graphs, heatmaps)
• Caching/lazy-loading: only fetch analysis data when requested
Architectural Patterns & Principles
• Layered architecture: Data pipelines → ML processing → API layer → Client UI
• Separation of concerns: distinct components for ingestion, processing, serving, visualization
• Caching to avoid redundant ML inference (.npy files)
• Lazy-loading and virtualization for performance
• Containerization: Docker Compose defines services and dependencies
Technologies & Frameworks
• Frontend: React, Vite, Material UI, Recharts, Nivo, Chart.js, react-force-graph-3d, react-window
• Backend: Flask, Python 3.10+, PyTorch, TensorFlow, Timm, Transformers, NumPy, OpenCV, Pillow
• External API: The Met Collection API
• Containers: Docker, Docker Compose