How ykato27/gemini-deep-search works
Project Type & Purpose
• This is a full-stack Python research-automation tool with two major surfaces:
– A CLI/batch pipeline that runs weekly (via GitHub Actions) to search, analyze, and generate Markdown reports.
– A Streamlit-based web dashboard for interactive exploration of historical data.
Main Components
A. Configuration Loader
– src/config_loader.py reads config.yaml (LLM models, Tavily settings, search parameters, email settings).
B. Data Collection & Analysis Pipeline
1. weekly_research.py (alias: app.py’s backend logic)
– Phase 1: research_searcher.py calls Tavily Search API + Google Gemini via LangGraph agent to collect raw articles → saves to reports/research_data.json
– Phase 2: research_analyzer.py uses Gemini to summarize and deep-analyze → emits weekly Markdown report under reports/
2. analyze_trends.py
– Aggregates all JSON files in reports/weekly_data → computes time-series trends → outputs to reports/trends/*.json
3. create_test_data.py
– Generates mock weekly_data files for testing
C. Notification Service
– email_report.py reads the latest Markdown report → connects to SMTP (Gmail App Password) → sends summary email to one or more recipients
D. Dashboard (Streamlit Frontend)
– app.py imports pages/1_* through pages/4_* as multi-page Streamlit UI
– Reads JSON and Markdown artifacts under reports/weekly_data and reports/trends to render:
• Home, Search, Trends, Companies views with interactive filters and charts (Plotly, pandas)
E. Orchestration / CI
– .github/workflows/weekly_research_split.yml triggers weekly GitHub Actions run
– Steps: Checkout → Set environment secrets → pip install → python weekly_research.py → python analyze_trends.py → commit reports → optionally run email_report.py
Data Stores & Formats
• config.yaml (YAML) – system parameters
• reports/research_data.json (raw API output)
• reports/weekly_data/YYYYMMDD.json (metadata + extracted insights)
• reports/trends/*.json (aggregated time-series)
• reports/週次レポート_YYYYMMDD.md (final report Markdown)
External Services
– Google Gemini API
– Tavily Search API
– Gmail SMTP (for email_report.py)
– GitHub Actions (for scheduling & CI)
Architectural Patterns & Design Principles
• ETL-style pipeline: Extract (searcher), Transform (analyzer), Load (reports folder)
• Modular separation of concerns: config loader, searcher, analyzer, trend processor, notifier, frontend
• File-based “database” for portability; JSON for intermediate, Markdown for reports
• CI/CD orchestration decouples scheduling from code logic
Relationships & Data Flows
– Config Loader → supplies parameters to Searcher & Analyzer
– Searcher → writes research_data.json → Analyzer
– Analyzer → writes weekly Markdown + weekly_data.json
– Trend Analyzer → reads weekly_data/*.json → writes trends/*.json
– Email Service → reads latest Markdown → external SMTP
– Dashboard → reads weekly_data + trends to serve UI
– GitHub Actions → triggers Pipeline & Trend Analyzer & Commit