# arockwell/local_newsifier architecture

> Architecture diagram of the GitHub repository arockwell/local_newsifier, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/arockwell/local_newsifier
- Repository: https://github.com/arockwell/local_newsifier

## Overview

##### Project type and purpose
- Local Newsifier is a monolithic “full-stack” backend service that exposes both a web/API component (FastAPI + Jinja2 UI templates) and a CLI (Typer), and it orchestrates asynchronous pipelines via Celery.
- Purpose: fetch local news articles (via RSS or Apify), perform NLP analyses (NER, headline trends, sentiment), track state, and persist results in PostgreSQL.

##### File structure
- Top-level entrypoints:
– src/local_newsifier/api → FastAPI app, routers, HTML templates
– src/local_newsifier/cli → Typer-based CLI commands
– src/local_newsifier/celery_app.py → Celery app instance
– scripts/*.py → one-off or demo scripts driving flows
- Core layers under src/local_newsifier:
– config → environment & database settings
– crud → ORM data-access layer
– models → Pydantic/SQLModel definitions
– services → business logic, orchestrating tools & CRUD
– tools → utility implementations (web_scraper, NER, trend_analyzer, etc.)
– flows → crew.ai flow definitions that chain services/tools
- Infra/config files: alembic.ini, Dockerfile, Procfile, Makefile, .github workflows, wheels directory for offline installs.

##### README insights
- Uses fastapi-injectable for dependency injection throughout API, services, tools, and flows.
- Integrates with Apify for web scraping external sites.
- Relies on Celery + a message broker (Redis or RabbitMQ) for scheduling and running asynchronous tasks.
- State persistence via PostgreSQL; migrations managed by Alembic.
- Observability: logging, error handling, workflow resumption.

##### Main components & their interactions
a. Presentation Layer
– CLI (Typer)
– Web/API (FastAPI + HTML templates)
b. Application Layer
– Dependency Injection container (fastapi-injectable providers)
– Services (analysis_service, rss_feed_service, apify_service, news_pipeline_service, etc.)
– Flows (crew.ai workflows: rss_scraping_flow, headline_trend_flow, entity_tracking_flow, etc.)
c. Infrastructure Layer
– Database: PostgreSQL (SQLModel/SQLAlchemy) + Alembic migrations
– Message Broker: Redis or RabbitMQ for Celery broker & result backend
– Celery Worker & Beat scheduler
– External Services: Apify (actor calls), spaCy NLP model
d. Data Flow & Dependencies
1. User invokes CLI command or submits HTTP request to FastAPI.
2. FastAPI handler / CLI command injects a Service via DI.
3. Service executes business logic: calls CRUD layer, tools, or triggers Flows.
4. For long-running or scheduled tasks, Flows enqueue Celery tasks.
5. Celery Beat schedules periodic Flow invocations; Celery Worker consumes tasks.
6. Worker tasks call Services and Tools (e.g., scrape, NER, trend analysis) and write results to DB.
7. Results are available via CLI/ API queries and UI templates.

##### Architectural patterns and design principles
- Hexagonal / Clean Architecture – UI (API/CLI) ↔ Application (Services/Flows) ↔ Infrastructure (DB, Broker, External).
- Dependency Injection – all components wired by fastapi-injectable providers for testability.
- Asynchronous Pipeline – Celery for durability, retry, scheduling.
- Flow Orchestration – crew.ai flows define multi-step workflows with state persistence.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Presentation Layer
    subgraph "Presentation Layer"
        direction TB
        UserShell(["User Shell"]):::external -->|CLI Command| CLI["Typer CLI"]:::internal
        UserBrowser(["User Browser"]):::external -->|HTTP Request| WebAPI["FastAPI (HTTP)"]:::internal
        CLI -->|invokes| CLIDependencies["CLI Dependencies"]:::internal
        WebAPI -->|uses| APIDependencies["API Dependencies"]:::internal
    end

    %% Application Layer
    subgraph "Application Layer"
        direction TB
        subgraph "Dependency Injection"
            direction TB
            DIInit["DI Module Init"]:::internal
            DIProviders["DI Providers"]:::internal
        end
        DIInit --> DIProviders

        subgraph "Services"
            direction TB
            AnalysisService["Analysis Service"]:::internal
            ApifyService["Apify Service"]:::internal
            ScheduleMgr["Apify Schedule Manager"]:::internal
            SourceConfigSvc["Apify Source Config Service"]:::internal
            ArticleService["Article Service"]:::internal
            EntityService["Entity Service"]:::internal
            PipelineService["News Pipeline Service"]:::internal
            RSSFeedService["RSS Feed Service"]:::internal
        end

        subgraph "Flows (crew.ai)"
            direction TB
            RSSFlow["RSS Scraping Flow"]:::async
            HeadlineFlow["Headline Trend Flow"]:::async
            TrendFlow["Trend Analysis Flow"]:::async
            EntityFlow["Entity Tracking Flow"]:::async
            NewsPipelineFlow["News Pipeline"]:::async
            PublicOpinionFlow["Public Opinion Flow"]:::async
        end

        subgraph "Tools / Utilities"
            direction TB
            WebScraper["Web Scraper"]:::internal
            RSSParser["RSS Parser"]:::internal
            EntityExtractor["Entity Extractor"]:::internal
            SentimentAnalyzer["Sentiment Analyzer"]:::internal
            SentimentTracker["Sentiment Tracker"]:::internal
            TrendAnalyzer["Trend Analyzer"]:::internal
            TrendReporter["Trend Reporter"]:::internal
            ContextAnalyzer["Context Analyzer"]:::internal
            EntityResolver["Entity Resolver"]:::internal
            EntityTracker["Entity Tracker Service"]:::internal
            OpinionVisualizer["Opinion Visualizer"]:::internal
            FileWriter["File Writer Utility"]:::internal
        end

        %% Internal interactions
        DIProviders -->|injects into| AnalysisService
        DIProviders --> ApifyService
        DIProviders --> ArticleService
        DIProviders --> EntityService
        DIProviders --> PipelineService
        DIProviders --> RSSFeedService

        AnalysisService -->|uses| EntityExtractor
        AnalysisService -->|uses| SentimentAnalyzer
        AnalysisService -->|uses| TrendAnalyzer
        RSSFeedService -->|uses| RSSParser
        ApifyService -->|uses| WebScraper
        PipelineService -->|triggers| RSSFlow
        PipelineService -->|triggers| NewsPipelineFlow

        RSSFlow -->|enqueues| CeleryBeat
        NewsPipelineFlow -->|enqueues| CeleryBeat
    end

    %% Infrastructure Layer
    subgraph "Infrastructure Layer"
        direction TB
        subgraph "Async Pipeline"
            direction TB
            CeleryApp["Celery App"]:::async
            CeleryTasks["Celery Tasks"]:::async
            CeleryBeat["Celery Beat Scheduler"]:::async
            CeleryWorker["Celery Worker"]:::async
        end

        subgraph "Database & Migrations"
            direction TB
            Postgres["PostgreSQL"]:::infra
            DBEngine["DB Engine"]:::infra
            Migrations["Alembic Migrations"]:::infra
        end

        subgraph "Message Broker"
            direction TB
            Broker["Redis / RabbitMQ"]:::infra
        end

        subgraph "External Services"
            direction TB
            Apify["Apify Actors"]:::external
            SpaCy["spaCy Model"]:::external
        end

        %% Infra interactions
        CeleryApp -->|loads| CeleryTasks
        CeleryBeat -->|schedules| CeleryWorker
        CeleryWorker -->|executes| CeleryTasks
        CeleryTasks -->|call| AnalysisService
        CeleryTasks -->|call| ApifyService
        CeleryTasks -->|call| PipelineService
        AnalysisService -->|persist| Postgres
        ArticleService -->|CRUD| DBEngine
        EntityService -->|CRUD| DBEngine
        RSSFeedService -->|CRUD| DBEngine
        Migrations -->|migrate| Postgres
        Broker -->|broker for| CeleryApp
        CeleryApp -->|broker| Broker
        ApifyService -->|calls| Apify
        AnalysisService -->|calls| SpaCy
    end

    %% Click Events
    click WebAPI "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/api/main.py"
    click APIDependencies "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/api/dependencies.py"
    click CLI "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/cli/main.py"
    click DIProviders "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/di/providers.py"
    click DIInit "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/di/__init__.py"
    click AnalysisService "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/services/analysis_service.py"
    click ApifyService "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/services/apify_service.py"
    click ScheduleMgr "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/services/apify_schedule_manager.py"
    click SourceConfigSvc "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/services/apify_source_config_service.py"
    click ArticleService "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/services/article_service.py"
    click EntityService "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/services/entity_service.py"
    click PipelineService "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/services/news_pipeline_service.py"
    click RSSFeedService "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/services/rss_feed_service.py"
    click RSSFlow "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/flows/rss_scraping_flow.py"
    click HeadlineFlow "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/flows/analysis/headline_trend_flow.py"
    click TrendFlow "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/flows/trend_analysis_flow.py"
    click EntityFlow "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/flows/entity_tracking_flow.py"
    click NewsPipelineFlow "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/flows/news_pipeline.py"
    click PublicOpinionFlow "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/flows/public_opinion_flow.py"
    click WebScraper "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/tools/web_scraper.py"
    click RSSParser "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/tools/rss_parser.py"
    click EntityExtractor "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/tools/extraction/entity_extractor.py"
    click SentimentAnalyzer "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/tools/sentiment_analyzer.py"
    click SentimentTracker "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/tools/sentiment_tracker.py"
    click TrendAnalyzer "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/tools/analysis/trend_analyzer.py"
    click TrendReporter "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/tools/analysis/trend_reporter.py"
    click ContextAnalyzer "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/tools/analysis/context_analyzer.py"
    click EntityResolver "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/tools/resolution/entity_resolver.py"
    click EntityTracker "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/tools/entity_tracker_service.py"
    click OpinionVisualizer "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/tools/opinion_visualizer.py"
    click FileWriter "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/tools/file_writer.py"
    click CeleryApp "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/celery_app.py"
    click CeleryTasks "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/tasks.py"
    click DBEngine "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/database/engine.py"
    click Migrations "https://github.com/arockwell/local_newsifier/tree/main/alembic/versions"
    click Broker "https://github.com/arockwell/local_newsifier/blob/main/scripts/init_celery_worker.sh"
    click Apify "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/services/apify_service.py"
    click SpaCy "https://github.com/arockwell/local_newsifier/blob/main/src/local_newsifier/config/settings.py"

    %% Styles
    classDef internal fill:#add8e6,stroke:#333,stroke-width:1px
    classDef async fill:#ffa500,stroke:#333,stroke-width:1px
    classDef external fill:#90ee90,stroke:#333,stroke-width:1px
    classDef infra fill:#d3d3d3,stroke:#333,stroke-width:1px
```
