How rahul-gaikwad-007/youtube-trend-analysis works
Main Components
• Web Interface (Frontend): Represented by app.py, this is the Streamlit-based UI that users interact with.
• YouTube Data Scraper: Indicated by brightdata_scrapper.py, this component uses the BrightData API to scrape video data from YouTube.
• AI Analysis Engine: Though no dedicated module file is provided for CrewAI, the README explains that CrewAI is used to analyze video transcripts and generate summaries. This should be shown as a separate processing component (you might call it the "CrewAI Analysis Module").
• Configuration & Environment: The config.yaml (along with .env instructions in the README) represents where configuration and secrets (e.g., the BrightData API key) are stored. Treat these as supporting components.
• External Services:
• BrightData: The external API service used for scraping YouTube videos.
• CrewAI (and its toolset): The external AI-based analysis service.
• Ollama (Llama 3.2 model): Although not directly in the file tree, the README instructions for installing and pulling the model indicate that it plays a role in the AI analysis process.
Relationships and Data Flow
• The user accesses the application via the Streamlit web interface (app.py).
• The web interface then calls the BrightData scraper (brightdata_scrapper.py) to fetch YouTube video data.
• The scraped data (including video transcripts) is then forwarded to the CrewAI Analysis Module, which may also interact with the Ollama-based model.
• Results (e.g., trend summaries or insights) are collected and rendered back to the user via the web interface.
• Configuration details (from config.yaml and .env) are used to securely manage API keys and other settings across the modules.
Architectural Patterns and Design Principles
• There is a clear separation of concerns:
• Frontend: Handles user interactions and presentation.
• Backend: Contains two main functions, data scraping and AI analysis.
Relevant Technologies and Frameworks
• Python 3.11+: The base programming language.
• Streamlit: For building the web interface.
• BrightData API: For accessing YouTube video data.
• CrewAI and crewai-tools: For transcript analysis and summary generation.
• Ollama: For deploying the Llama 3.2 model, which may underpin parts of the AI analysis process.
• YAML and environment management: config.yaml and .env files for setup and configuration.
• Each component should be clearly labeled (e.g., “Streamlit Frontend (app.py)”, “BrightData Scraper (brightdata_scrapper.py)”, “CrewAI Analysis Module”, “Config & Environment”).
• Data flow from user input to scraping and then to analysis.
• Interaction with external services (BrightData API, CrewAI, and Ollama).
• External services (e.g., rounded rectangles or differently shaded components).