How wmoto-ai/elyza-tasks-100-humanevaluator works
Project Type and Purpose
• Project Type: This is a full-stack web application.
• Purpose: The system collects user responses to the Elyza Tasks 100 questions, sends these responses for grading by GPT-4o via the OpenAI API, displays the grading results along with feedback, and then compares these scores with results from other AI models. The outcomes are further visualized using graphs, and all data (user responses and grading results) is stored in a CSV file.
File Structure
• Top-level Elements: The repository includes configuration files (Dockerfile, docker-compose.yml), environment setup files (.gitignore, requirements.txt, LICENSE), and key directories such as app, data, and images.
• app Directory: Contains the main application code (app/main.py) and static web assets (app/static/index.html). This suggests that the application’s user interface is served as a static HTML page, with the backend logic implemented in Python.
• data Directory: Although it contains elyza_tasks_100.csv as a dataset, the README mentions that user responses and grading results are appended to data (likely data/results.csv). This indicates that file-based persistence is used in place of a traditional database.
• Configuration and Deployment: The Dockerfile and docker-compose.yml suggest that the application can be containerized for deployment, and setup instructions also detail running the app directly via Python.
README for Additional Insights
• The README explains the application’s flow:
• A user enters their username and begins the evaluation.
• The system presents a question (from the Elyza Tasks 100 dataset).
• User responses are graded by GPT-4o and compared against scores from other AI models.
• Results, feedback, and graph visualizations are rendered on the front-end.
• Dependencies: Python 3.8+, Docker, and an OpenAI API key are required.
• External Services: Communication with OpenAI’s API (for GPT-4o) is integral, and results from other AI models are referenced for comparison.
a. Main Components:
• Frontend:
• Static User Interface (served from app/static/index.html).
• Input forms for username and answers.
• Graphical display components to show grading results and comparisons.
• Backend:
• Python Application (app/main.py) – managing request handling, processing submission data, API communication, and business logic.
• Data Storage:
• CSV Storage in the “data” directory (for saving user responses and grading results).
• Dataset File (elyza_tasks_100.csv) that stores the questions.
• External Service:
• OpenAI API (GPT-4o for scoring responses).
• Other reference AI model scores (HODACHI/EZO-Humanities-9B-gemma-2-it, GPT-4, Claude 3.5 Sonnet) which are integrated by fetching or comparing the data.
• Containerization/Deployment:
• Docker Container (Dockerfile) and Docker Compose orchestration (docker-compose.yml).
b. Relationships and Interactions:
• The Browser (Frontend) sends user input (username, answers) to the Python backend.
• The Backend processes the input and issues API calls outward to the OpenAI API (for GPT-4o grading).
• Results returned from the OpenAI API are combined with local business logic (e.g., comparing scores) and saved to the CSV file in the data directory.
• The Backend sends processed data back to the frontend, where the UI updates to show textual feedback and graphs.
• Deployment Unit: The entire system runs within Docker containers (if deployed via Docker), with the container(s) encapsulating both the Python environment and any dependencies.
c. Architectural Patterns/Design Principles:
• Layered Architecture: Clear separation between the user interface (frontend) and backend service logic.
• File-based Persistence: Instead of using a full database, results and data are stored in CSV, which is appropriate for lightweight data volumes.
• Containerization: Promotes a reproducible and isolated runtime environment via Docker.
• API-Based Communication: Interfacing with external APIs (OpenAI API) as key integration points.
d. Key Technologies, Frameworks, or Libraries:
• Python: The main programming language (with any underlying web framework not explicitly indicated, potentially Flask or similar).
• Docker and Docker Compose: For containerized deployment.
• OpenAI API: For GPT-4o to perform scoring.
• Static HTML/CSS/JS: For building the user interface.
• CSV file handling libraries in Python: To manage data storage.
Elements
• Clear Labels: Each component (e.g., “User Browser”, “Static UI”, “Python Backend”, “OpenAI API”, “CSV Data Store”, “Docker Container”) should be clearly labeled.
• Another distinct style for external services/external integrations (APIs).
Final Recommendations
• Be as detailed as possible – ensure that every component, its role, and every interaction (data flow, API calls, updates to the CSV store) is represented.
• Do not shy away from breaking the project into many components; capturing all architectural layers (frontend, backend, external communications, persistence, containerization) will provide accurate and useful insights.
• This modular approach not only aids clarity but also helps in future troubleshooting and further architectural refinements.