How sanuvishwakarma/agenticai_url_summarizer works
Project Type and Purpose
• This is a full-stack AI Pipeline application with:
• A lightweight frontend (Streamlit).
• A Python backend orchestrating a LangGraph “agent” pipeline.
• Multiple processing “nodes” (fetch, extract, summarize).
• A vector database for embeddings (ChromaDB).
• External AI services (open-source LLMs & embedding models).
Main Components
• Frontend: Streamlit UI (app.py)
• Orchestrator: LangGraph agent runner (main.py)
• Processing Nodes:
• fetch_url_content
• extract_content
• summarize_content
• Vector Store Abstraction: vector_store.py
• Database: ChromaDB (chroma_db folder, sqlite + binary files)
• External Services:
• HTTP/Web (requests)
• BeautifulSoup (HTML parsing)
• HuggingFace Llama-1.1B (text generation)
• Sentence-Transformers (embedding model)
Component Relationships & Data Flow
1. User enters a URL in Streamlit → “Summarize” button triggers:
2. app.py calls main.py to run the LangGraph pipeline.
3. main.py initializes agent and passes URL to the first node.
4. fetch_url_content node:
• Requests URL
• Downloads raw HTML
↓
5. extract_content node:
• Parses HTML with BeautifulSoup
• Extracts clean text
↓
6. summarize_content node:
• Sends text prompt to Llama-1.1B
• Receives summary
↓
7. vector_store.py:
• Embeds summary (Sentence-Transformers)
• Stores embedding + metadata in ChromaDB
8. main.py collects summary → returns to app.py
9. Streamlit displays the summary (and optionally retrieval results)
Architectural Patterns & Design Principles
• Modular Pipeline: Each node is a single responsibility.
• Separation of Concerns: UI vs. orchestration vs. processing vs. storage.
• Extensibility: New nodes can be dropped into the nodes/ folder.
• Local stateful storage for embeddings (ChromaDB) decouples compute from persistence.
• Use of open-source models (HuggingFace, Sentence-Transformers) and standard libraries.