How lala850307-glitch/fin-intel-query-core works
This repo is a multi-surface product, not a single app.
It contains four major subsystems: a web application, an offline desktop package, a data ingestion / ETL pipeline, and a search/AI enrichment toolkit. The architecture is centered on a shared company dataset that is built offline and then served through either PostgreSQL or SQLite depending on deployment mode.
Web runtime: FastAPI backend + React frontend.
The live product uses app.py as the Python backend entrypoint and frontend/ as a Vite + React + TypeScript SPA. The frontend holds the landing page, search UI, company detail pages, admin pages, and API registration flows. The backend likely exposes the search, detail, tracking, update, and sync endpoints described in the README, with Uvicorn as the serving runtime.
Search is the core application boundary.
Search is not a single query path; it combines multiple strategies: exact match, fuzzy match, pinyin, zhuyin-to-Chinese conversion, alias matching, and semantic/vector search. The repo has dedicated search implementations under search/ for online/PostgreSQL scenarios and exe/search_company_sqlite.py for offline SQLite mode. This means the search layer is split by deployment target but shares the same conceptual scoring model.
The dataset is assembled through a multi-step ETL pipeline.
Business data is built from scripts in scripts/ and clean_scripts/, with a documented sequence: base company list, GCIS enrichment, alias generation, financials, pricing, and flagship table assembly. The pipeline can run step-by-step or via run_pipeline.py, with run_daily.sh and run_quarterly.sh orchestrating scheduled execution. The produced “flagship” table appears to be the main denormalized search/index dataset consumed by the app.
There are specialized crawler and enrichment sub-pipelines.
langchain/finance_report/ and langchain/langchain/pipeline/ indicate separate ingestion flows for news, IR reports, sentiment, summarization, and CSV export. These outputs feed the intelligence layer of the product rather than the base company database. The repo also includes embedding generation artifacts under Embedding/, suggesting offline preprocessing for vector search or similarity tasks.
Semantic search depends on external model infrastructure.
The README references Ollama with bge-m3:latest and pgvector. That means semantic retrieval is not just an application feature; it is a dependency on a local or server-side model runtime and a vector-capable database schema. In the offline package, this dependency is deliberately excluded, which keeps the packaged app self-contained.
User tracking and API products are a separate service layer.
Beyond company search, the backend also manages Google OAuth login, anonymous/session tracking, API key issuance, and usage logging. The data model described in the README (users, user_events, tab_durations, api_applications, api_usage_logs, app_download_logs) indicates a second domain focused on product analytics and paid/controlled API access rather than search itself.
The API exposure layer is versioned and metered.
The public API is under /api/v1 and supports search, company detail, related companies, and popular queries. API keys are generated server-side, expire after 48 hours, and every call is logged with query, latency, and IP. This is a clear boundary between the public API product and the internal search implementation.
Offline desktop packaging is a first-class distribution path.
The exe/ directory contains PyInstaller specs, a bundle launcher, SQLite export tooling, and platform-specific build assets. The desktop app starts a local backend, opens a native window via pywebview, auto-selects a free port, and uses local database snapshots. That means the “app” is actually a bundled client-server system running on the user’s machine.
The offline app has its own update/sync control plane.
At startup it checks GitHub Releases for a new executable and checks the server for a newer data snapshot. The README describes background download/sync plus front-end polling for update readiness. This creates two asynchronous control flows: one for binary updates and one for data freshness, both independent of the search UI.
Automation and deployment are externalized.
ETL is not cron-driven inside the repo; it is triggered by an external OpenClaw scheduler over SSH. Telegram notifications are used for step-level progress and failure reporting. CI/CD is handled by GitHub Actions release workflows that build Mac and Windows artifacts from tagged releases.
Frontend structure is organized by user journey and admin concerns.
Under frontend/src/pages/, the app is split into Landing, Home, Search, CompanyDetail, Team, ApiRegister, and a substantial Admin console. The Company Detail page is further divided into tabs for board, branch, financial, news, related companies, and stock data, reflecting the backend’s denormalized company-centric model.
External company/financial/market sources are ingested by the scripts, normalized and enriched, optionally vectorized, and then written to PostgreSQL or SQLite. The search backend reads from that store and serves both the web frontend and the public API, while the desktop build uses the same pipeline output exported into a local SQLite snapshot.
Main entry points: App.tsx Frontend · app.py Backend · 06_build_flagship.py ETL scripts · search_company_sqlite.py SQLite search
Components 21
Web app
FastAPI backend, React SPA, and public API surface
- Backend · Main Python entrypoint for the live web/backend runtime
app.py - Frontend · Vite + React TypeScript single-page application
frontend/src/App.tsx - Search UI · Search, landing, and company browsing experience
frontend/src/pages/Search/index.tsx - Company detail · Company profile view with board, branch, financial, news, related, and stock tabs
frontend/src/pages/CompanyDetail/index.tsx
Desktop app
Packaged offline client-server runtime with local SQLite
- SQLite snapshot · Embedded database shipped with the desktop build
exe/search_data.db - Desktop bundle · Packaged local client-server launcher with webview shell
exe/app_bundle.py - SQLite search · Local search implementation for the desktop package
exe/search_company_sqlite.py - SQLite export · Exports the built dataset into the embedded desktop database
exe/export_to_sqlite.py
ETL pipeline
Batch enrichment and flagship dataset assembly
- ETL scripts · Stepwise enrichment scripts for base list, GCIS, aliases, financials, pricing, and flagship build
scripts/06_build_flagship.py - Clean pipeline · Normalization and intermediate cleanup stages used by the ETL flow
clean_scripts/clean_06_flagship.py - Schedulers · Shell orchestration for daily and quarterly execution
run_pipeline.py
Search and AI
Deployment-specific search engines and semantic enrichment
- Online search · Primary search implementation for the live PostgreSQL-backed product
search/search_company_completwithAI_sql_only.py - Search fallback · Alternate SQL-backed search path with reduced AI dependency
search/search_company_completwithAI_sql_fallback.py - Semantic assets · Alias expansion, zhuyin/pinyin handling, and embedding artifacts for semantic retrieval
search/build_company_alias_ollama_db.py - Ollama model · Local or server-side embedding model used for semantic search
Tracking and API ops
User analytics, API keys, and usage metering
- Admin console · Operational dashboards for users, API, downloads, analytics, and settings
frontend/src/pages/Admin/index.tsx - API register · Frontend flow for API application and key issuance
frontend/src/pages/ApiRegister/index.tsx - Tracking API · Frontend-side event and usage reporting
frontend/src/api/track.ts
Data stores
Shared source, intermediate, and published datasets
- Flagship data · Published company-centric CSV used by search and product views
data/flagship/06_taiwan_company_flagship.csv - Raw sources · Imported upstream source files used by ETL and enrichment
data/raw/factory.csv - PostgreSQL store · Primary live database with vector and product-analytics tables
Connections 23
- Frontend → Search UI: routes to · App shell renders the search journey
- Frontend → Company detail: routes to · App shell renders company-centric detail pages
- Frontend → Admin console: routes to · App shell renders operational dashboards
- Frontend → API register: routes to · App shell renders API signup flow
- Frontend → Tracking API: reports · Client events and dwell-time telemetry are sent from the UI
- Backend → Frontend: serves · Backend hosts the web application and API consumers
- Backend → Online search: queries · Live backend delegates search requests to the online engine
- Backend → PostgreSQL store: reads/writes · Backend persists users, usage, and searchable company data
- Online search → PostgreSQL store: reads · Primary search executes against PostgreSQL-backed tables and vectors
- Online search → Semantic assets: uses · Search relies on alias expansion, pinyin/zhuyin normalization, and embeddings
- Semantic assets → Ollama model: depends on · Embedding and similarity flows rely on external model runtime
- Search fallback → PostgreSQL store: reads · Reduced-dependency search path still queries the main online store
- ETL scripts → Raw sources: ingests · Batch pipeline starts from raw source files
- Clean pipeline → ETL scripts: feeds · Cleanup stages prepare enriched inputs for the main ETL build
- ETL scripts → Flagship data: produces · Pipeline assembles the denormalized flagship company dataset
- Schedulers → ETL scripts: orchestrates · Scheduled runs trigger the staged enrichment pipeline
- SQLite export → Flagship data: exports from · Desktop packaging copies the built dataset into SQLite form
- SQLite export → SQLite snapshot: writes · Creates the embedded snapshot shipped with the offline app
- Desktop bundle → SQLite search: launches · Packaged app starts the local search backend
- SQLite search → SQLite snapshot: reads · Offline search runs entirely against the embedded database
- SQLite search → Flagship data: mirrors · Offline mode is derived from the same flagship dataset lineage
- Backend → Tracking API: logs · Backend consumes usage and telemetry events for analytics
- Backend → API register: issues keys · Backend manages API applications and credentials