How optas/artemis works
Project Type & Purpose
• ArtEmis is a research codebase focused on affective language for visual art. It is designed to enable data preprocessing, analysis, emotion-classifier training, and neural speaker generation for painting captioning and emotion analysis.
• The project encompasses data processing (preprocessing and cleaning), exploratory analysis (via notebooks), and training pipelines for deep learning models that associate emotions to images and text.
Analyzing the File Structure
• Top-level directories and files:
• The main package is “artemis” which contains submodules including “analysis”, “captioning”, “evaluation”, “in_out”, “language”, “neural_models”, “utils”, and “data”. These denote clear modular separation.
• “notebooks” are provided for exploratory analysis, training experiments, and evaluation.
• The “scripts” directory encapsulates command-line interfaces for preprocessing, training, sampling, and evaluation.
• Each submodule focuses on a distinct aspect:
• “analysis”: Contains statistical and linguistic analysis scripts (e.g., emotion-centric and feature extraction functions).
• “captioning” and “neural_models”: Focus on building captioning models and neural network components for attention, encoder/decoder, and classification.
• “evaluation”: Includes metrics implementations (BLEU, METEOR, CIDEr, etc.) and evaluation scripts to critique the generated captions.
• “in_out”: Manages data input/output operations such as reading, cleaning, and manipulating datasets.
• “language”: Provides functions for linguistic processing (e.g., part-of-speech tagging, adjective-noun pair extraction, spell checking).
• “utils”: Houses common utilities (basic functions, vocabulary management, visualization).
Key Architectural Components
a. Data & Preprocessing Layer:
• Input Data: Raw ArtEmis dataset (CSV, images, pre-trained vocabulary files, etc.) located under “artemis/data”.
• Preprocessing Scripts: Found in “artemis/scripts/preprocess_artemis_data.py” and “artemis/in_out”. These handle cleaning, tokenization, spell-checking, data splits, and creating input files for modeling.
b. Analysis & Exploration Layer:
• Analysis modules in “artemis/analysis” and Jupyter notebooks under “artemis/notebooks/analysis” for linguistic and statistical analysis (e.g., emotion histogram extraction and entropy analysis).
c. Modeling & Training Layer:
• Neural Models: Located in “artemis/neural_models” (attentive decoders, image and text classifiers, etc.) and “artemis/language” (LSTM encoders, transformer-based modules).
• Training Scripts/Notebooks:
• Emotion-centric training notebooks (image-to-emotion, utterance-to-emotion) and scripts (e.g., “train_speaker.py” in “artemis/scripts”).
• Pretrained models are provided externally, with instructions in the README on how to integrate them.
d. Caption Generation (Speaker) & Evaluation:
• Captioning Module: Under “artemis/captioning” includes sample captions and sentiment analysis with sentiment-aware algorithms.
• Evaluation: “artemis/evaluation” includes various metrics (BLEU, METEOR, CIDEr, etc.) and notebooks for evaluating generated captions with aspects like emotional alignment.
e. External Dependencies & Configuration:
• The project relies on third-party libraries (PyTorch, CUDA, etc.) as specified in setup.py.
• The README describes environment setup and execution commands, which are integrated in the pipeline.
Relationships and Data Flow
• The Analysis layer (using notebooks) for exploratory analysis.
• The Modeling layer where training routines (via scripts and notebooks) consume preprocessed data to train emotion classifiers and neural speakers.
• After model training, predictions/captions can be generated (using sampling scripts) and then evaluated using the “Evaluation” module.
• External pretrained models (image-to-emotion, text emotion classifiers, SAT-speaker) act as inputs or dependencies for some of these pipelines.
Architectural Patterns & Principles
• Modularity: The codebase is clearly divided into self-contained modules addressing specific functionalities (preprocessing, analysis, modeling, evaluation).
• Separation of Concerns: Different directories handle distinct operational aspects such as neural models, language operations, and in/out operations.
• Reusability: Functions from “utils” and “language” can be reused across different parts of the system.
• Critical modules are represented as separate components with clear labels.
• Loop-backs or feedback cycles (like retraining based on evaluation outcomes).