# decodingml/second-brain-ai-assistant-course architecture

> Architecture diagram of the GitHub repository decodingml/second-brain-ai-assistant-course, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/decodingml/second-brain-ai-assistant-course
- Repository: https://github.com/decodingml/second-brain-ai-assistant-course

## Overview

##### Project Type and Its Purpose
- This project is an open-source course and codebase for building a “Second Brain AI Assistant” that leverages agents, large language models (LLMs), retrieval-augmented generation (RAG), and LLMOps best practices.
- It comprises two primary Python applications:
- An offline pipeline (apps/second-brain-offline) handling data ingestion, ETL processes, vector indexing, crawling, dataset generation, and other ML pipelines.
- An online pipeline (apps/second-brain-online) that implements the agentic inference engine used as the live AI assistant.
- In addition, there is supporting Docker and infrastructure configuration (apps/infrastructure) and workshop templates with code examples.

##### File Structure
- Top-Level Directories:
- .github: Contains CI/CD workflows which indicate a continuous integration pipeline.
- apps: Contains subdirectories for “infrastructure”, “second-brain-offline”, and “second-brain-online.”
- Offline pipelines (“second-brain-offline”) are organized into:
- Configurations (.env, YAML, pyproject.toml)
- Pipeline scripts (under pipelines/ and steps/) for various stages such as collection, ETL, vector indexing, and dataset generation.
- A full source tree (under src/second_brain_offline) that adheres to a layered structure including application (agents, crawlers, dataset generators, models, RAG components), domain logic, and infrastructure (integration with AWS S3, MongoDB, and Notion).
- Online inference (“second-brain-online”) follows a similar structure, featuring agents, evaluation, and RAG components embedded in an operational real-time API.
- The infrastructure directory inside apps/infrastructure contains Docker-related files – important for deployment and container orchestration.

##### Extract Additional Insights from the README
- The README explains that the course is divided into modules covering both offline ML pipelines and online AI assistant functionality.
- It specifically highlights advanced RAG with contextual/parent retrieval, dataset generation via distillation, fine-tuning of LLMs, and agent building.
- Several images provided (system_architecture.png, rag_feature_pipeline_architecture.png, agentic_rag_architecture.png, etc.) set the stage for a multi-component and multi-layered architecture.

##### Components
a. Main Components:
- Offline Pipeline Application:
- Data Collection & Crawling: Notion API crawler, web crawling agents (e.g., crawl4ai) that collect and preprocess data.
- ETL & Dataset Generation: Pipelines that clean data, add quality scores, generate fine-tuning datasets, and compute vector indices using either OpenAI or Hugging Face endpoints.
- RAG Pipeline: Components that manage retrieval, embeddings (vector indexing), and splitting documents for answering queries.
- Online Pipeline Application (Second Brain AI Assistant):
- Inference Engine: Real-time agentic inference service where agents integrate LLMs with retrieval components.
- Agents & Evaluation: Modules handling decision-making, summarization, quality evaluation, and other agent tools.
- API Layer: The service endpoint that accepts user queries and returns responses using the agentic RAG approach.
- Supporting Infrastructure:
- Docker Containerization: Infrastructure setup with Dockerfile and docker-compose configuration facilitating containerized deployment.
- Cloud Integrations: Connections to external services such as AWS S3 for storage, MongoDB for document indexing and retrieval, and Notion for initial data ingestion.
- CI/CD Pipeline: GitHub workflows to automate tests and deployments.

b. Relationships and Interactions:
- Data Flow: Start at data ingestion (Notion, crawling) → processed in offline pipelines (ETL, dataset generation, vector indexing) → data stored in MongoDB/S3 or other databases → consumed by online pipeline during inference.
- Control Flow: The offline pipelines are triggered either via Makefiles or scheduled CI tasks. The online AI assistant interacts via API endpoints, and agents internally call retrieval and inference services.

c. Architectural Patterns and Principles:
- Layered Architecture: Separate concerns between application logic (agents, crawlers, RAG components), domain models, and infrastructure (cloud storage, databases).
- Modular Pipelines: Breaking down different tasks (collection, preprocessing, retrieval, inference) into independent modules/pipelines.
- Containerization & CI/CD: Utilizing Docker for deployment consistency and GitHub Actions for automated testing and builds.
- Microservices/Cohesion in Design: Although both offline and online components are part of the same repository, treat them as separate self-contained services.

d. Technologies, Frameworks, and Libraries:
- Python (as seen from .py files, pyproject.toml, and the code structure).
- Docker (Dockerfile, docker-compose.yml).
- Notion API integrations.
- AWS S3 and MongoDB for storage and data retrieval.
- LLM providers (OpenAI, Hugging Face) based on different configuration YAML pipelines.
- Tools such as Makefiles, uv (modern Python tooling and linters like ruff).

##### Essential Elements
- Application modules (e.g., offline pipelines vs. online service).
- Infrastructure/configuration components versus processing components.
- External service integrations.

- Data Ingestion → Crawling & Collection Modules → ETL & Preprocessing Pipelines → RAG Pipeline (Embedding & Vector Indexing) → Storage (MongoDB, S3)
- On the right side, for the Online Pipeline:
- API / Inference Endpoint → Agentic Components (Agents, Evaluators, Summarizers) which then query the storage or directly consume precomputed vector indices → Calls to external LLM providers.
- Optionally, include a section representing the workshop/template artifacts to illustrate how learners can replicate or extend the architecture.

Happy diagramming!

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Offline Pipeline Subgraph
    subgraph "Offline Pipeline"
        O1["Offline Pipeline Application"]:::offline
        O2["Notion Data Collector"]:::offline
        O3["Web Crawler"]:::offline
        O4["ETL & Dataset Generation"]:::offline
        O5["RAG Pipeline"]:::offline
    end

    %% Storage and External Integration Subgraph
    subgraph "Storage & External Integration"
        S1["AWS S3 Storage"]:::external
        S2["MongoDB Storage"]:::external
        E1["Notion Integration"]:::external
        LLM["LLM Providers(OpenAI/HuggingFace)"]:::external
    end

    %% Online Pipeline Subgraph
    subgraph "Online Pipeline"
        ON1["Online Pipeline Application"]:::online
        ON2["API Inference Endpoint"]:::online
        ON3["Agentic Inference Engine"]:::online
    end

    %% Supporting Infrastructure Subgraph
    subgraph "Supporting Infrastructure"
        INF["Docker & CI/CD Infrastructure"]:::infra
    end

    %% Workshops & Templates Subgraph
    subgraph "Workshops & Templates"
        W1["Workshop Solutions"]:::workshop
        W2["Workshop Templates"]:::workshop
        W3["Additional Learning Materials"]:::workshop
    end

    %% Connections for Offline Pipeline
    O2 -->|"feeds"| O4
    O3 -->|"feeds"| O4
    O4 -->|"transforms"| O5
    O5 -->|"indexes"| S1
    O5 -->|"indexes"| S2
    O2 -->|"integrates"| E1

    %% Connections for Online Pipeline
    ON1 -->|"exposes"| ON2
    ON2 -->|"routes_to"| ON3
    ON3 -->|"queries"| S1
    ON3 -->|"queries"| S2
    ON3 -->|"calls"| LLM

    %% Connecting Supporting Infrastructure
    O1 --> INF
    ON1 --> INF

    %% Connections to Workshops (illustrative learning paths)
    O1 -.->|"learning"| W1
    W1 -->|"guides"| W2
    W2 -->|"supplements"| W3
    ON1 -.->|"learning"| W3

    %% Class Definitions for Colors
    classDef offline fill:#f9e79f,stroke:#7d6608,stroke-width:2px;
    classDef online fill:#aed6f1,stroke:#1b4f72,stroke-width:2px;
    classDef infra fill:#d5f5e3,stroke:#196f3d,stroke-width:2px;
    classDef external fill:#fadbd8,stroke:#b03a2e,stroke-width:2px;
    classDef workshop fill:#d6eaf8,stroke:#2874a6,stroke-width:2px;

    %% Click Events for Offline Pipeline
    click O1 "https://github.com/decodingml/second-brain-ai-assistant-course/tree/main/apps/second-brain-offline"
    click O2 "https://github.com/decodingml/second-brain-ai-assistant-course/blob/main/apps/second-brain-offline/pipelines/collect_notion_data.py"
    click O3 "https://github.com/decodingml/second-brain-ai-assistant-course/blob/main/apps/second-brain-offline/src/second_brain_offline/application/crawlers/crawl4ai.py"
    click O4 "https://github.com/decodingml/second-brain-ai-assistant-course/blob/main/apps/second-brain-offline/pipelines/etl.py"
    click O5 "https://github.com/decodingml/second-brain-ai-assistant-course/blob/main/apps/second-brain-offline/pipelines/compute_rag_vector_index.py"

    %% Click Event for External Integrations (Offline)
    click E1 "https://github.com/decodingml/second-brain-ai-assistant-course/tree/main/apps/second-brain-offline/src/second_brain_offline/infrastructure/notion"
    click S1 "https://github.com/decodingml/second-brain-ai-assistant-course/blob/main/apps/second-brain-offline/src/second_brain_offline/infrastructure/aws/s3.py"
    click S2 "https://github.com/decodingml/second-brain-ai-assistant-course/tree/main/apps/second-brain-offline/src/second_brain_offline/infrastructure/mongo"

    %% Click Events for Online Pipeline
    click ON1 "https://github.com/decodingml/second-brain-ai-assistant-course/tree/main/apps/second-brain-online"
    click ON3 "https://github.com/decodingml/second-brain-ai-assistant-course/tree/main/apps/second-brain-online/src/second_brain_online/application/agents"

    %% Click Event for Supporting Infrastructure
    click INF "https://github.com/decodingml/second-brain-ai-assistant-course/tree/main/apps/infrastructure/docker"

    %% Click Events for Workshops & Templates
    click W1 "https://github.com/decodingml/second-brain-ai-assistant-course/tree/main/workshops/rag/solution"
    click W2 "https://github.com/decodingml/second-brain-ai-assistant-course/tree/main/workshops/rag/template"
    click W3 "https://github.com/decodingml/second-brain-ai-assistant-course/tree/main/workshops/workshops_data"
```
