# camtice/automated_modeling architecture

> Architecture diagram of the GitHub repository camtice/automated_modeling, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/camtice/automated_modeling
- Repository: https://github.com/camtice/automated_modeling

## Overview

##### Main Components

- Framework Core: The heart of the application is the modular “inspect_ai” package under the src/inspect_ai directory. This is further divided into submodules such as:
– _cli: The command-line interface functionality that provides tools for evaluation, logging, and sandbox task execution.
– _eval: The evaluation engine that handles tasks like running evaluations, handling evaluation sets and logging outcomes.
– _model and _providers: Interfaces and implementations for accessing different large language models (from OpenAI, Hugging Face, Azure AI, Anthropic, etc.).
– _display and _view: Modules that support both textual/rich output and a full web interface (for example, the “textual” displays and the web UI in src/inspect_ai/_view/www) for model output and result visualization.
– approval, dataset, scorer, solver, and tool: Additional feature-specific modules that implement prompt approval workflows, dataset management, scoring mechanics, solving tasks, and tool integrations.
– _util: Utility functions that support common tasks (e.g., configuration, asynchronous processing, environment/sandbox management).

- External Interfaces and Tools:
– CLI Executables (e.g. app.py and inspect_ai/__main__.py) to bootstrap the application.
– The web-based user interface components (within _view, the “www” subdirectory, and the tools/vscode integration directories), enabling a visual representation of logs and tasks.
– External integrations (model providers, tool definitions, and API clients) that connect to various LLM services.

- Experimentation & Data Management:
– The “modeling” directory holds experiments, raw data, configuration files, and CSV results.
– The “tests” and “examples” directories reveal how the evaluation engine is verified and how plug-in components are used in practice.

##### Relationships and Interactions

- Data Flow: Show how input (e.g. model prompts, evaluation tasks from CLI or web UI) enters the system, then flows through components:
– The CLI layer gathers commands (via _cli), which trigger evaluations (_eval).
– These requests further call model functions (_model and its provider implementations) and then feed results to the scoring component (_scorer) and display modules (_display/_view).
– Logging and transcript management (under _log) capture output and events, while _util components help in file, configuration, and error management.

- Experimentation and Testing: Represent the “modeling” experiments and the “tests” suites as supporting layers that feed back into the core by validating functionality and performance.

##### Architectural Patterns and Design Principles

- Modularity: Emphasize that the project is partitioned into loosely coupled modules (via separate directories such as _cli, _eval, _model, _display, and others). This modularity allows for extensibility (for adding new scoring techniques or tool integrations) and separate testing.
- Abstraction: The “providers” subdirectory under _model abstracts the details of calling different external APIs, ensuring that core functionality is provider agnostic.
- Plug-In Extensibility: Indicate that extensions (via separate Python packages) can integrate additional evaluation or scoring techniques.

##### Relevant Technologies and Frameworks
- Python is the primary language with dependency management through pip and pyproject.toml.
- The CLI uses standard Python scripting practices (managed via app.py and __main__.py).
- Web UI components leverage modern JavaScript tooling (via the “www” directory) along with frameworks that support rich text and textual displays (like Rich and Textual).
- The testing framework uses a combination of pytest and pre-commit hooks.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interface Layer
    subgraph "User Interface Layer"
        UInput["User Input"]:::input
        CLI["CLI Interface (_cli)"]:::cli
        WebUI["Web UI (Display)"]:::web
        Entry1["Main Executable (app.py)"]:::entry
        Entry2["Package Entrypoint (__main__)"]:::entry
    end

    %% Core Engine Layer
    subgraph "Core Engine Layer"
        Core["Framework Core (inspect_ai)"]:::core
        Eval["Evaluation Engine (_eval)"]:::eval
        Util["Utility Functions (_util)"]:::util
        Model["Model Abstraction (_model)"]:::model
        Providers["Model Providers (_providers)"]:::modelp
    end

    %% Feature Modules Layer
    subgraph "Feature Modules"
        Approval["Approval Module"]:::feature
        Dataset["Dataset Module"]:::feature
        Scorer["Scoring Module"]:::feature
        Solver["Solver Module"]:::feature
        Tool["Tool Integrations"]:::feature
    end

    %% Display and Output Layer
    subgraph "Display/Logging Layer"
        Textual["Textual/Rich Display (_display)"]:::display
        WebDisplay["Web UI Display (_view/www)"]:::display
    end

    %% Experimentation & Data Management Layer
    subgraph "Experimentation & Data Management"
        Modeling["Experiments/Data (modeling)"]:::data
        Tests["Testing (tests)"]:::data
        Examples["Examples (examples)"]:::data
    end

    %% Connections from User Interface to Core Engine
    UInput --> CLI
    UInput --> WebUI
    CLI --> Eval
    WebUI --> Eval
    CLI --> Entry1
    CLI --> Entry2

    %% Core Engine internal connections
    Core --> Eval
    Core --> Model
    Core --> Util
    Eval --> Util
    Eval --> Model
    Model --> Providers

    %% Evaluation Engine interacting with Feature Modules
    Eval --> Approval
    Eval --> Dataset
    Eval --> Scorer
    Eval --> Solver
    Eval --> Tool

    %% Evaluation results sent to Display Layer
    Eval --> Textual
    Eval --> WebDisplay

    %% Logging/Experimentation flow
    Eval --> Modeling
    Tests --> Core
    Examples --> Core

    %% Click Events
    click Core "https://github.com/camtice/automated_modeling/tree/main/src/inspect_ai"
    click CLI "https://github.com/camtice/automated_modeling/tree/main/src/inspect_ai/_cli"
    click Eval "https://github.com/camtice/automated_modeling/tree/main/src/inspect_ai/_eval"
    click Model "https://github.com/camtice/automated_modeling/tree/main/src/inspect_ai/model"
    click Providers "https://github.com/camtice/automated_modeling/tree/main/src/inspect_ai/model/_providers"
    click Textual "https://github.com/camtice/automated_modeling/tree/main/src/inspect_ai/_display"
    click WebDisplay "https://github.com/camtice/automated_modeling/tree/main/src/inspect_ai/_view/www"
    click Approval "https://github.com/camtice/automated_modeling/tree/main/src/inspect_ai/approval"
    click Dataset "https://github.com/camtice/automated_modeling/tree/main/src/inspect_ai/dataset"
    click Scorer "https://github.com/camtice/automated_modeling/tree/main/src/inspect_ai/scorer"
    click Solver "https://github.com/camtice/automated_modeling/tree/main/src/inspect_ai/solver"
    click Tool "https://github.com/camtice/automated_modeling/tree/main/src/inspect_ai/tool"
    click Util "https://github.com/camtice/automated_modeling/tree/main/src/inspect_ai/_util"
    click Entry1 "https://github.com/camtice/automated_modeling/blob/main/app.py"
    click Entry2 "https://github.com/camtice/automated_modeling/blob/main/src/inspect_ai/__main__.py"
    click Modeling "https://github.com/camtice/automated_modeling/tree/main/modeling"
    click Tests "https://github.com/camtice/automated_modeling/tree/main/tests"
    click Examples "https://github.com/camtice/automated_modeling/tree/main/examples"

    %% Styles
    classDef input fill:#f9c,stroke:#333,stroke-width:2px;
    classDef cli fill:#bbf,stroke:#333,stroke-width:2px;
    classDef web fill:#bbf,stroke:#333,stroke-width:2px;
    classDef entry fill:#ddd,stroke:#333,stroke-width:2px;
    classDef core fill:#faa,stroke:#333,stroke-width:2px;
    classDef eval fill:#ffa,stroke:#333,stroke-width:2px;
    classDef util fill:#cfc,stroke:#333,stroke-width:2px;
    classDef model fill:#fcf,stroke:#333,stroke-width:2px;
    classDef modelp fill:#fcf,stroke:#333,stroke-width:2px;
    classDef feature fill:#aff,stroke:#333,stroke-width:2px;
    classDef display fill:#ddf,stroke:#333,stroke-width:2px;
    classDef data fill:#fdd,stroke:#333,stroke-width:2px;
```
