How mysterionrise/agentbricks-experiments works
Project type & purpose (what you’re diagramming)
This repo is not a “traditional app” with services and source packages; it’s a minimal Databricks workspace demo to stand up two Agent Bricks workflows:
1. Knowledge Assistant (KA) backed by Unity Catalog (UC) Volume files generated by a notebook.
2. Custom LLM backed by a toy labeled UC table (request, response) generated by a notebook.
File structure → architectural components
From the tree:
• notebooks/00_check_prereqs.py
• Validates workspace/runtime prerequisites (notably embedding endpoint availability: databricks-gte-large-en).
• notebooks/01_setup_catalog_schema_volume.py
• Creates/ensures UC catalog + schema + volume (default main.ab_demo.ka_data).
• notebooks/02_load_sample_docs_to_volume.py
• Writes markdown docs into the UC Volume path:
/Volumes/<catalog>/<schema>/<volume>/knowledge
• notebooks/03_build_custom_llm_dataset.py
• Creates UC table main.ab_demo.custom_llm_tiny with (request,response) examples.
• notebooks/04_query_knowledge_assistant_example.py
• Shows how to query the deployed agent endpoint via MLflow Deployments.
Other files:
Architecturally, the notebooks are “infrastructure + data bootstrapping scripts,” not a runtime service.
README → key platform interactions you must capture
Important constraints and flows called out in the README:
• Requires Unity Catalog, Serverless compute, and access to foundation models via system.ai.
• Knowledge Assistant knowledge source must be:
• UC Volume files OR Vector Search index (not used in this demo, but mention as an optional branch).
• Knowledge Assistant depends on an embedding model endpoint: databricks-gte-large-en.
• Creation/deployment is done through Databricks UI (Agents → Knowledge Assistant / Custom LLM), after data is prepared by notebooks.
• Querying happens via AI Playground or MLflow Deployments (programmatic client).
User / Developer
• Runs notebooks in Databricks workspace
• Creates agents in Databricks UI
• Queries endpoints from Playground or code
Databricks Workspace
• Notebooks (the repo content executed on compute)
• Serverless Compute / Cluster (execution environment for notebooks)
Unity Catalog (Governance layer)
• Catalog + Schema
• UC Volume: main.ab_demo.ka_data
• Folder: /knowledge containing markdown files
• UC Table: main.ab_demo.custom_llm_tiny (custom LLM dataset)
Agent Bricks (Agents service)
• Knowledge Assistant Agent
• Knowledge source: UC Files (volume path ending in /knowledge)
• Custom LLM Agent
• Dataset: UC table with request/response columns
• Deployed Agent Endpoints (the runtime endpoints produced after build/sync/optimization)
Model/Embedding Services
• Embedding endpoint: databricks-gte-large-en (required for KA ingestion/indexing)
• Foundation model access via system.ai (used by the agent runtime)
Client Query Layer
• AI Playground (interactive testing)
• MLflow Deployments client (used in notebook 04_... to call the endpoint)
Optional branch (call out as “not used in this repo but supported”):
• Vector Search Index as an alternative knowledge source to UC Files.
Flow 0: Prerequisite check
• 00_check_prereqs.py → checks workspace context + availability of databricks-gte-large-en
Flow 1: Knowledge Assistant data preparation
• 01_setup_catalog_schema_volume.py:
• 02_load_sample_docs_to_volume.py:
Flow 2: KA agent build/sync + deployment (control plane action)
• User in Agents → Knowledge Assistant:
Flow 3: Custom LLM dataset + agent deployment
• 03_build_custom_llm_dataset.py:
• User in Agents → Custom LLM:
Flow 4: Querying
• From Playground:
• From code:
• 04_query_knowledge_assistant_example.py:
Architectural patterns to highlight (explicitly label these)
• Notebook-driven provisioning: notebooks act as “IaC-lite” for UC objects and sample data.
• Governed data plane: all knowledge sources live in Unity Catalog (Volumes/Tables).
• Managed agent lifecycle: build/sync/optimization/deploy handled by Databricks Agent Bricks (control plane).
• RAG-style retrieval for Knowledge Assistant:
• UC Files → embeddings (databricks-gte-large-en) → managed retrieval → LLM response.
• Endpoint-based serving:
• Both agents ultimately expose a deployed endpoint consumed by Playground or MLflow client.
• Databricks Unity Catalog (catalog/schema/volume/table)
• UC Volumes path convention /Volumes/<catalog>/<schema>/<volume>/...
• Agent Bricks Knowledge Assistant + Custom LLM
• Embeddings: databricks-gte-large-en
• LLM access: system.ai
• Query client: MLflow Deployments
• Execution: Databricks Serverless compute (or the compute mode you run notebooks on)