How microsoft/genaiops-azureaisdk-template works
Project Type and Purpose
• This is a full-stack AI experiment accelerator designed to manage end-to-end generative AI experiments; it covers local experimentation, evaluation, and automated deployment via GitHub Actions.
• It integrates local development environments (with Python/Conda setups), CI/CD using GitHub workflows, and infrastructure provisioning using Terraform for Azure.
Analyzing the File Structure
• Top-level directories such as:
– .github: Contains GitHub Actions workflows for CI/CD, experiment validations, and infrastructure deployments.
– docs: Documentation and guides covering setup, execution, and best practices.
– infra: Terraform scripts and Azure CLI scripts for provisioning Azure infrastructure (storage, AI services, etc.).
– lib: Reusable code like custom evaluators.
– llmops: Core scripts for running experiments.
– math_coding and math_coding_agent: Use case folders that encapsulate separate flows, evaluations, deployments, and data.
– tests: Unit tests for validating main scripts.
• The clear separation into “use case” folders (each containing subdirectories for data, flows, evaluations, and deployment) indicates an architecture that supports scalable experimentation for multiple AI use cases.
Key Architectural Components
a. Frontend/Interface & Local Development:
– The local environment integration (VS Code, Conda setup) and documentation explain that users interact via a local development environment.
b. Experimentation Engine:
– Components in llmops and use case folders (math_coding, math_coding_agent) manage experiment logic, configurations, and flow definitions.
– The experiment YAML files define the core experiment’s orchestration – including flow definitions, evaluation functions, and environment variables.
c. Evaluation & Monitoring:
– Custom evaluators in lib and different evaluators in the use case folders handle offline and online evaluation.
– There is integrated observability where logs, metrics, and evaluations are routinely tracked.
d. CI/CD and Orchestration:
– GitHub Actions workflows (inside .github/workflows) automate testing (PR validations) and deployment (dev workflows).
– Workflows for infra deployment automate provisioning through Terraform scripts located in infra/terraform.
e. Infrastructure & External Integration:
– Infra folder contains scripts (Terraform, Azure CLI) that set up Azure resources (e.g., Function Apps, Storage, Key Vault).
– Integration with Azure AI Foundry and Azure AI services is done via pre-configured connections specified in the experiment YAML.
Relationships and Interactions
– Local Development Environment → Experiment Engine (llmops/math_coding)
– Experiment Engine → GitHub Actions (pr and dev workflows triggering validations and deployments)
– Deployment Pipelines → Infrastructure (via Terraform in the infra folder)
– Experiment Engine accesses external services (Azure AI Foundry via defined connections in experiment YAML) using environment variables for credentials.
• Reusability aspects (e.g., the shared evaluator components in lib that serve multiple use cases).
Recommended Elements
• Use clear labels for each component:
• “Local Development Environment”
• “Experiment Engine” (with inner sub-components for flows, evaluations, and experiments)
• “CI/CD Pipelines (GitHub Actions)” – differentiate PR validation and Dev workflows
• “Azure Infrastructure” – Terraform-based deployment and Azure AI services
• “External Services” – AI Agents, OpenAI connections, and monitoring/logging endpoints.
• How data (e.g., dataset inputs, evaluation results) flows from experiment initiation to AI service calls.
• Deployment triggers, such as pushing code to GitHub initiating CI/CD workflows.
• Rectangles for components (folders/modules).