How ed-donner/sds works
Project type and purpose
• This repo is not a traditional full-stack application but rather a curriculum repository for an 8-week “SuperDataScience AI Bootcamp.” It contains:
• Course materials (Jupyter notebooks under week1/…, cohort1/week1/…, etc.)
• Knowledge-base markdown files (in knowledge-base/ folders)
• Python scripts for labs, ingestion, evaluation, and agentic-AI demos
• Setup and troubleshooting guides
• Its “system” is essentially a content-and-code delivery platform for students, featuring sample pipelines that call OpenAI APIs.
File structure
Top-level:
• cohort1/, week1/, week2/, … — multiple parallel folder structures holding weekly materials.
• guides/ — standalone Jupyter-based tutorials on Git, Python, Async, etc.
• setup/ — environment installation and troubleshooting scripts.
• assets/ — static images (e.g., banner).
• pyproject.toml, uv.lock, .python‐version — project configuration and dependencies.
Within each week or cohort week folder:
• lab*.ipynb or .py — hands-on exercises
• knowledge-base/ — markdown content for company, contracts, employees, products
• Supporting Python modules: e.g., week2’s ingest.py, embeddings.py, evaluator.py, week3’s deep_research agents.
Extract architecture hints from the README
• Purpose: guide through LLMs, Agents, and Production AI systems.
• It mentions “making API calls to the OpenAI API,” “deploying AI systems for Production,” and points students at setup and troubleshooting.
• Ingest markdown “knowledge bases”
• Generate embeddings
• Query LLMs
• Evaluate results
• Orchestrate multi-agent workflows.
Main components
a. Content & Code Repositories
• Knowledge-base files (markdown)
• Lab notebooks (Jupyter)
• Python modules (ingest, embeddings, evaluator, agent scripts)
• Guides & setup docs
b. Execution Environment
• Student workstation/IDE (VS Code with Cursor)
• Python runtime + venv (managed by pyproject.toml and .python-version)
• Jupyter Server
c. AI Pipeline Subsystems
• Ingestion service (scripts read markdown)
• Embedding generator (calls OpenAI embeddings API)
• LLM query service (calls Chat/Completion endpoints)
• Evaluation engine (tests against tests.jsonl)
• Multi-Agent orchestrator (deep_research agents)
d. External Services
• OpenAI API
• (Optionally) Vector store or local cache for embeddings
• GitHub for version control
Relationships & data flow
• Student triggers a lab notebook →
1. Notebook imports Python modules (ingest.py, embeddings.py, etc.)
2. Ingest reads markdown files → passes raw text →
3. Embedding generator sends text to OpenAI embeddings endpoint → receives vectors →
4. Notebook or evaluation script issues queries to LLM via OpenAI completions API →
5. Evaluation module compares LLM output against ground-truth (tests.jsonl) → feedback.
• In week3 deep_research: Planner Agent → Research Manager → Search Agent / Push Agent / Writer Agent loop.
Architectural patterns & design principles
• Modular: each pipeline stage is its own Python module or notebook.
• Separation of concerns: ingestion, embedding, querying, evaluation all decoupled.
• Agent orchestration: follows “manager–worker” pattern for multi-agent experiments.
• Configuration as code: dependency management via pyproject.toml and lockfiles.
• Content as data: knowledge-base markdown separated from code.
Detail
• Don’t just lump “labs” together—break out major pipelines week by week.
• Multi-agent workflow separately.
• Configuration files (pyproject.toml, .python-version) feeding into the environment layer.
• Precisely mirrors this bootcamp’s repo structure and content-code separation
• Illustrates the AI pipelines students implement each week
• Displays interactions with external APIs and environment setup