How gurpreetkaurjethra/ai-lead-generation-agent works
Project type and purpose
• This is a single-script, full-stack AI agent application with a Streamlit front end that automates lead discovery, qualification, and storage.
• It orchestrates multiple external services (Firecrawl’s Search & Extract APIs, OpenAI GPT-4o, Composio Google Sheets) via phidata.
File structure
• Top‐level files only:
• ai_lead_generation_agent.py – the monolithic Streamlit app + orchestration logic
• requirements.txt – lists Firecrawl SDK, phidata, composio, streamlit, openai, etc.
• IMG_AILG – folder of architecture/UX screenshots
• No separate backend directory: all components are invoked from within the script.
Key insights from the README
• Streamlit is used for the web UI (“streamlit run …”).
• phidata is used for “agent orchestration” – i.e., managing the workflow of search → extract → qualification → sheet update.
• Firecrawl provides two endpoints: search (to find Quora URLs) and extract (to get profile data).
• Composio integrates with Google Sheets to persist leads.
• OpenAI GPT-4o is used to qualify extracted profiles.
Main components and their interactions
a. Components:
1. Streamlit UI
2. Agent Orchestrator (phidata)
3. Firecrawl Search API
4. Firecrawl Extract API
5. OpenAI GPT-4o Qualification
6. Composio Google Sheets Integration
7. Google Sheets (external data store)
b. Relationships & data flow:
• User → Streamlit UI: enters search criteria & starts run
• Streamlit UI → phidata agent: triggers workflow
• Agent → Firecrawl Search API: retrieves list of Quora URLs
• Agent → Firecrawl Extract API (in loop): fetches profile data for each URL
• Agent → OpenAI GPT-4o: passes raw profile data for qualification & enrichment
• Agent → Composio SDK: writes formatted, qualified leads to Google Sheets
• Google Sheets ← Composio: stores final lead table
• Streamlit UI polls or displays progress / final results
c. Architectural patterns:
• Orchestrator (phidata) implements a pipeline/workflow pattern
• Monolith + external service integration rather than microservices
• Reactive UI (Streamlit) calls into orchestration
d. Key technologies:
• Streamlit for UI
• phidata for workflow orchestration
• Firecrawl SDK & REST APIs for web scraping
• OpenAI Python SDK for GPT-4o calls
• Composio Python SDK for Sheets integration
• Google Sheets as persistent storage
• Blue rectangle for Streamlit UI
• Green circle/hexagon for phidata Orchestrator
• Orange ellipses for external APIs (Firecrawl Search, Firecrawl Extract, OpenAI)
1. User → Streamlit
2. Streamlit → phidata orchestrator
3. phidata → Firecrawl Search API → returns URLs
4. phidata → Firecrawl Extract API → returns profiles
5. phidata → OpenAI → returns qualifiers
6. phidata → Composio → writes to Google Sheets
7. Google Sheets → final persisted leads
Detailed capture of essential elements
• Expected data formats (JSON response from Firecrawl, tabular rows to Sheets).
• Clearly separates UI, orchestration, service integrations, and data store
• Shows the precise flow through Firecrawl, OpenAI, and Composio
• Highlights the core design principle: a workflow-driven, fully automated lead generation agent