How momenashra/agentic-patterns works
Project type and purpose
• This is an open-source demonstration repository showcasing two related patterns for LLM-driven agents:
a. A Reflection Pattern (in Reflection-pattern/) that implements an agent capable of self-reflection and iterative improvement.
b. An Autonomous Multi-Agent Web Scraping Framework (in multiagent-pattern/ plus the top-level scripts described in the README) that orchestrates Search, Scrape, Aggregate, and Reporting agents to collect and summarize product data.
File structure
• Top-level files (per README, even if not shown in your tree):
• run_pipeline.py – launches the multi-agent crew
• tasks.py – defines the Searcher, Scraper, Aggregator, and Reporter agents
• tools.py – custom utilities (e.g., JSON writer)
• output/ – stores JSON results and generated reports
• Reflection-pattern/ sub-directory:
• reflection_pattern.ipynb – demonstration notebook
• src/reflection_agent.py – core agent logic implementing reflection loops
• utils/completions.py & utils/logging.py – abstraction over LLM calls and logging
• multiagent-pattern/ sub-directory:
• Notebooks (Product_searcher.ipynb, step_5_enhanced_report.html) showing stages of the scraping pipeline
README for additional insights
• Describes four pipeline phases: Search → Scrape → Aggregate → Report
• Mentions use of CrewAI for orchestration and external services (Cohere, AgentOps) for LLM and tooling
• Emphasizes modularity (agents & tools can be swapped), scalability (parallel execution), and reproducibility
a. Main components
• Reflection Agent Subsystem
• LLM Interface (via completions.py)
• Reflection Loop Controller (reflection_agent.py)
• Logging/Telemetry (logging.py)
• Multi-Agent Pipeline Subsystem
1. Search Agent – queries target sites
2. Scraper Agent – extracts page data
3. Aggregator Agent – cleans & merges JSON
4. Writer Agent – generates human-readable report
• Orchestration Layer
• CrewAI orchestrator (in run_pipeline.py and tasks.py)
• External Services
• Cohere API
• AgentOps API
• Data Stores / Outputs
• Intermediate JSON files (output/)
• Final HTML/Markdown/PDF reports
b. Relationships & interactions
• User invokes run_pipeline.py → CrewAI initializes agent instances
• Search Agent calls Cohere (optional NLP) + target websites → returns URL list
• URL list flows to Scraper Agent, which writes raw JSON
• Raw JSON flows to Aggregator Agent → outputs cleaned JSON
• Clean JSON flows to Writer Agent → emits report files
• Reflection Agent (in isolation) interacts with LLM via completions.py, loops on its own output for refinement
c. Architectural patterns & principles
• Pipeline Pattern: clear stage separation, data flows in a directed acyclic graph
• Modular Agents: each agent is a micro-component with a single responsibility
• Orchestration via CrewAI: pluggable execution order (parallel vs. sequential)
• Extensibility: tools and agents can be swapped without changing core orchestration
d. Key technologies / libraries
• Python 3.10+
• CrewAI (pipeline orchestration)
• Tavily-Python, ScrapeGraph-Py for scraping helpers
• Cohere API, AgentOps API for LLM calls
• Standard Python modules (logging, JSON, argparse)