How blue-jays/news_bias_check_companion works
Project Type and Purpose
• This is a data-driven full-stack system for ingesting, analyzing, and bias-scoring news articles, with an accompanying browser extension UI, a REST API backend, ML feature pipelines, orchestrated data workflows, data validation, and monitoring/observability.
Main Architectural Components
a. Browser Extension
• Files under extension/ (manifest.json, content_script.js, popup & options UI)
• Provides user interface for fetching or displaying bias scores in the browser.
b. API Service (Backend)
• FastAPI application in src/app/ (routers, service modules, DB layer, telemetry)
• Exposes endpoints for health checks, article analysis, source management.
c. Database
• SQL database accessed via src/app/db/*.py and configured in configs/app.yaml
• Stores articles, sources, analysis results.
d. CLI Tool
• src/cli/main.py for ad-hoc commands or dev workflows.
e. ML Pipelines
• Extraction, feature generation, bias-ranking models under src/ml/
• Topic extraction, sentiment features, vectorization, model inference.
f. Orchestrators & Pipelines
• src/orchestrators/actions/nightly_fetch.py triggers scheduled tasks.
• src/pipelines/{ingest,preprocess,register,evaluate}.py define ETL/data-science steps.
g. Data Validation
• Great Expectations under validation/great_expectations/ for schema and data-quality checks.
h. Infrastructure & Monitoring
• infra/prometheus.yml, infra/grafana/ dashboards collect metrics from API and pipelines.
i. Configuration & Deployment
• Dockerfile, compose.yaml for local or containerized deployment.
• GitHub Actions CI and nightly‐fetch jobs in .github/workflows/.
Component Relationships and Data Flows
• Browser Extension → (HTTPS) → API Service → (SQL queries) → Database
• Orchestrator Scheduler → Python scripts (pipelines) →
• Fetch articles from external news sources (configs/sources.yaml)
• Preprocess & load into Database → Validate with Great Expectations →
• Feature extraction & model inference → Write results back to Database
• API Service invokes ML service modules for on-demand analysis.
• Monitoring agents scrape metrics from API and pipeline processes → Prometheus → Grafana dashboards.
• CLI Tool may directly invoke pipelines or API for testing/dev.
Architectural Patterns & Principles
• Separation of concerns: UI, API, data, ML, orchestration, monitoring are discrete.
• Modular ML: feature extraction, drift detection, bias scoring are pluggable under src/ml/.
• Configuration-driven deployment: YAML files in configs/, compose.yaml.
• Observability by default: logging, metrics, dashboards.