How arockwell/local_newsifier works
Project type and purpose
• Local Newsifier is a monolithic “full-stack” backend service that exposes both a web/API component (FastAPI + Jinja2 UI templates) and a CLI (Typer), and it orchestrates asynchronous pipelines via Celery.
• Purpose: fetch local news articles (via RSS or Apify), perform NLP analyses (NER, headline trends, sentiment), track state, and persist results in PostgreSQL.
File structure
• Top-level entrypoints:
– src/local_newsifier/api → FastAPI app, routers, HTML templates
– src/local_newsifier/cli → Typer-based CLI commands
– src/local_newsifier/celery_app.py → Celery app instance
– scripts/*.py → one-off or demo scripts driving flows
• Core layers under src/local_newsifier:
– config → environment & database settings
– crud → ORM data-access layer
– models → Pydantic/SQLModel definitions
– services → business logic, orchestrating tools & CRUD
– tools → utility implementations (web_scraper, NER, trend_analyzer, etc.)
– flows → crew.ai flow definitions that chain services/tools
• Infra/config files: alembic.ini, Dockerfile, Procfile, Makefile, .github workflows, wheels directory for offline installs.
README insights
• Uses fastapi-injectable for dependency injection throughout API, services, tools, and flows.
• Integrates with Apify for web scraping external sites.
• Relies on Celery + a message broker (Redis or RabbitMQ) for scheduling and running asynchronous tasks.
• State persistence via PostgreSQL; migrations managed by Alembic.
• Observability: logging, error handling, workflow resumption.
Main components & their interactions
a. Presentation Layer
– CLI (Typer)
– Web/API (FastAPI + HTML templates)
b. Application Layer
– Dependency Injection container (fastapi-injectable providers)
– Services (analysis_service, rss_feed_service, apify_service, news_pipeline_service, etc.)
– Flows (crew.ai workflows: rss_scraping_flow, headline_trend_flow, entity_tracking_flow, etc.)
c. Infrastructure Layer
– Database: PostgreSQL (SQLModel/SQLAlchemy) + Alembic migrations
– Message Broker: Redis or RabbitMQ for Celery broker & result backend
– Celery Worker & Beat scheduler
– External Services: Apify (actor calls), spaCy NLP model
d. Data Flow & Dependencies
1. User invokes CLI command or submits HTTP request to FastAPI.
2. FastAPI handler / CLI command injects a Service via DI.
3. Service executes business logic: calls CRUD layer, tools, or triggers Flows.
4. For long-running or scheduled tasks, Flows enqueue Celery tasks.
5. Celery Beat schedules periodic Flow invocations; Celery Worker consumes tasks.
6. Worker tasks call Services and Tools (e.g., scrape, NER, trend analysis) and write results to DB.
7. Results are available via CLI/ API queries and UI templates.
Architectural patterns and design principles
• Hexagonal / Clean Architecture – UI (API/CLI) ↔ Application (Services/Flows) ↔ Infrastructure (DB, Broker, External).
• Dependency Injection – all components wired by fastapi-injectable providers for testability.
• Asynchronous Pipeline – Celery for durability, retry, scheduling.
• Flow Orchestration – crew.ai flows define multi-step workflows with state persistence.