# ykato27/gemini-deep-search architecture

> Architecture diagram of the GitHub repository ykato27/gemini-deep-search, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ykato27/gemini-deep-search
- Repository: https://github.com/ykato27/gemini-deep-search

## Overview

##### Project Type & Purpose
- This is a full-stack Python research-automation tool with two major surfaces:
– A CLI/batch pipeline that runs weekly (via GitHub Actions) to search, analyze, and generate Markdown reports.
– A Streamlit-based web dashboard for interactive exploration of historical data.

##### Main Components
A. Configuration Loader
– src/config_loader.py reads config.yaml (LLM models, Tavily settings, search parameters, email settings).
B. Data Collection & Analysis Pipeline
1. weekly_research.py (alias: app.py’s backend logic)
– Phase 1: research_searcher.py calls Tavily Search API + Google Gemini via LangGraph agent to collect raw articles → saves to reports/research_data.json
– Phase 2: research_analyzer.py uses Gemini to summarize and deep-analyze → emits weekly Markdown report under reports/
2. analyze_trends.py
– Aggregates all JSON files in reports/weekly_data → computes time-series trends → outputs to reports/trends/*.json
3. create_test_data.py
– Generates mock weekly_data files for testing
C. Notification Service
– email_report.py reads the latest Markdown report → connects to SMTP (Gmail App Password) → sends summary email to one or more recipients
D. Dashboard (Streamlit Frontend)
– app.py imports pages/1_* through pages/4_* as multi-page Streamlit UI
– Reads JSON and Markdown artifacts under reports/weekly_data and reports/trends to render:
- Home, Search, Trends, Companies views with interactive filters and charts (Plotly, pandas)
E. Orchestration / CI
– .github/workflows/weekly_research_split.yml triggers weekly GitHub Actions run
– Steps: Checkout → Set environment secrets → pip install → python weekly_research.py → python analyze_trends.py → commit reports → optionally run email_report.py

##### Data Stores & Formats
- config.yaml (YAML) – system parameters
- reports/research_data.json (raw API output)
- reports/weekly_data/YYYYMMDD.json (metadata + extracted insights)
- reports/trends/*.json (aggregated time-series)
- reports/週次レポート_YYYYMMDD.md (final report Markdown)

##### External Services
– Google Gemini API
– Tavily Search API
– Gmail SMTP (for email_report.py)
– GitHub Actions (for scheduling & CI)

##### Architectural Patterns & Design Principles
- ETL-style pipeline: Extract (searcher), Transform (analyzer), Load (reports folder)
- Modular separation of concerns: config loader, searcher, analyzer, trend processor, notifier, frontend
- File-based “database” for portability; JSON for intermediate, Markdown for reports
- CI/CD orchestration decouples scheduling from code logic

##### Relationships & Data Flows
– Config Loader → supplies parameters to Searcher & Analyzer
– Searcher → writes research_data.json → Analyzer
– Analyzer → writes weekly Markdown + weekly_data.json
– Trend Analyzer → reads weekly_data/*.json → writes trends/*.json
– Email Service → reads latest Markdown → external SMTP
– Dashboard → reads weekly_data + trends to serve UI
– GitHub Actions → triggers Pipeline & Trend Analyzer & Commit

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Orchestration
    GH["GitHub Actions"]:::orchestrator

    %% Batch Pipeline
    subgraph "Batch Pipeline"
        CL["Config Loader"]:::process
        CS["Research Searcher"]:::process
        CA["Research Analyzer"]:::process
        AT["Trend Aggregator"]:::process
        TD["Test Data Generator"]:::process
    end

    %% Notification Service
    ER["Email Reporter"]:::process

    %% Frontend Dashboard
    UI["Streamlit UI"]:::frontend

    %% Data Stores
    CFG["config.yaml"]:::datastore
    RAW["reports/research_data.json"]:::datastore
    WDATA["reports/weekly_data/"]:::datastore
    TDATA["reports/trends/"]:::datastore
    MD["reports/週次レポート_*.md"]:::datastore

    %% External Services
    GEM["Google Gemini API"]:::external
    TAV["Tavily Search API"]:::external
    SMTP["Gmail SMTP"]:::external

    %% Relationships
    GH -->|triggers| CS
    GH -->|triggers| AT
    GH -->|optional email| ER

    CS -->|"writes"| RAW
    CS -->|uses| CL
    CS -->|calls| TAV
    CS -->|calls| GEM

    CA -->|reads| RAW
    CA -->|writes| WDATA
    CA -->|writes| MD
    CA -->|uses| CL
    CA -->|calls| GEM

    AT -->|reads| WDATA
    AT -->|writes| TDATA

    TD -->|writes mock files to| WDATA

    ER -->|reads| MD
    ER -->|sends via| SMTP

    UI -->|reads| WDATA
    UI -->|reads| TDATA

    CL -->|reads| CFG

    %% Click Events
    click CL "https://github.com/ykato27/gemini-deep-search/blob/main/src/config_loader.py"
    click CFG "https://github.com/ykato27/gemini-deep-search/blob/main/config.yaml"
    click CS "https://github.com/ykato27/gemini-deep-search/blob/main/src/research_searcher.py"
    click RAW "https://github.com/ykato27/gemini-deep-search/blob/main/reports/research_data.json"
    click CA "https://github.com/ykato27/gemini-deep-search/blob/main/src/research_analyzer.py"
    click MD "https://github.com/ykato27/gemini-deep-search/blob/main/reports/週次レポート_*.md"
    click WDATA "https://github.com/ykato27/gemini-deep-search/tree/main/reports/weekly_data/"
    click AT "https://github.com/ykato27/gemini-deep-search/blob/main/analyze_trends.py"
    click TDATA "https://github.com/ykato27/gemini-deep-search/tree/main/reports/trends/"
    click TD "https://github.com/ykato27/gemini-deep-search/blob/main/create_test_data.py"
    click ER "https://github.com/ykato27/gemini-deep-search/blob/main/email_report.py"
    click UI "https://github.com/ykato27/gemini-deep-search/blob/main/app.py"
    click UI "https://github.com/ykato27/gemini-deep-search/blob/main/pages/1_🏠_Home.py"
    click UI "https://github.com/ykato27/gemini-deep-search/blob/main/pages/2_🔍_Search.py"
    click UI "https://github.com/ykato27/gemini-deep-search/blob/main/pages/3_📈_Trends.py"
    click UI "https://github.com/ykato27/gemini-deep-search/blob/main/pages/4_🏢_Companies.py"
    click UI "https://github.com/ykato27/gemini-deep-search/blob/main/.streamlit/config.toml"
    click GH "https://github.com/ykato27/gemini-deep-search/blob/main/.github/workflows/weekly_research_split.yml"

    %% Styles
    classDef process fill:#E8F1F2,stroke:#1B4F72,stroke-width:2px
    classDef datastore fill:#FDEBD0,stroke:#B55318,stroke-width:2px,stroke-dasharray: 5 5
    classDef external fill:#D5F5E3,stroke:#196F3D,stroke-width:2px,stroke-dasharray: 2 2
    classDef orchestrator fill:#EBDEF0,stroke:#6C3483,stroke-width:2px
    classDef frontend fill:#D6EAF8,stroke:#1B4F72,stroke-width:2px,stroke-dasharray: 4 2
```
