# rimamedia/newsloom architecture

> Architecture diagram of the GitHub repository rimamedia/newsloom, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rimamedia/newsloom
- Repository: https://github.com/rimamedia/newsloom

## Overview

##### Project Type and Purpose
- NewLoom is a full-stack, Django-based news aggregation and publishing platform.
- It combines three pillars:
– A Django web backend (REST API, admin, WebSockets)
– A custom “stream scheduler” for fetching and processing content
– An integration layer (Playwright, Slack, Telegram, Articlean)

##### File Structure
- Top-level orchestration & infra:
– Dockerfiles (Dockerfile, Dockerfile.ecs, Dockerfile.frontend)
– docker-compose.yml / docker-compose.frontend.yml
– Nginx configs (dev-nginx.conf, nginx.conf, nginx.frontend.conf, nginx_ecs.conf)
– Supervisord configs (supervisord.conf, supervisord.docker.conf)
- Documentation: /docs (Sphinx source)
- Django monorepo structure:
– newsloom/ (project settings, Celery config, ASGI/WGSI entrypoints)
– apps: agents, chat, frontend (API & serializers), mediamanager, sources, streams
– manage.py and custom management commands (e.g., run_streams)
- Static & build scripts: build-frontend.sh, FRONTEND_SETUP.md
- Dependencies & CI/CD: buildspec.yml, GitHub Actions workflows

##### Extract Key Architecture Notes from README
- Backend: Django 5.1.3, DRF, Channels
- Scheduler: management command `run_streams`, tracks execution stats
- Integrations: Playwright (JS-heavy scraping), Slack & Telegram bots, Articlean API
- Storage: PostgreSQL (or SQLite), Redis (Channels layer)
- Deployment: Docker + Nginx for static/media, Docker-Compose or ECS

a. Client Side
– Browser SPA or direct REST/WebSocket clients
b. Edge & Load Balancer
– Nginx (HTTP & WS proxy, static file server)
c. Web Tier
– Gunicorn/Uvicorn workers running Django ASGI for HTTP & Channels
d. Scheduler Tier
– A separate container/process executing `python manage.py run_streams`
– Optionally managed by supervisord or an ECS task
e. Data & Cache
– PostgreSQL cluster
– Redis instance (Channels layer & optional Celery broker)
f. Integration Services
– Playwright headless browsers
– External APIs: Slack, Telegram, Articlean
g. Storage Volumes
– Docker volumes for media and static files

##### Relationships & Interactions
- Browser → Nginx (HTTP/HTTPS) → Gunicorn/Uvicorn → Django REST endpoints
- Browser → Nginx (WS) → Django Channels → chat_consumer/message events
- Django stores/fetches data from PostgreSQL and uses Redis for pub/sub
- Scheduler process → Streams services → external sources → writes to DB
- On new data, scheduler (or Django signals) publishes updates via Channels to clients
- Django admin UI uses the same web tier & DB
- Playwright runs inside the scheduler container when fetching JS-heavy pages
- Slack/Telegram integrations invoked by stream tasks or chat consumers

##### Architectural Patterns & Design Principles
- Monolithic Django codebase split into reusable “apps” (agents, chat, sources, streams)
- Layered design: Web/API layer vs. Scheduler layer vs. Integration layer
- Asynchronous messaging: Django Channels + Redis
- Custom pub/sub within Django for real-time notifications
- Management commands as lightweight micro-processes for scheduling
- Docker-first deployment: each component in its own container

##### Final Touches
- Readability at both high level (overall layout) and detail level (individual flows)

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Client Side"
        direction TB
        Browser["Browser SPA"]:::client
    end
    click Browser "https://github.com/rimamedia/newsloom/tree/main/newsloom/frontend/"

    subgraph "Edge & Load Balancer"
        direction TB
        Nginx["Nginx Proxy\n(HTTP/HTTPS 80/443, WS)"]:::edge
    end
    click Nginx "https://github.com/rimamedia/newsloom/blob/main/nginx.conf"

    subgraph "Django Monolith"
        direction TB

        subgraph "Web Tier"
            direction TB
            WebContainer["Django ASGI Web Container\n(Gunicorn/Uvicorn)"]:::web
        end
        click WebContainer "https://github.com/rimamedia/newsloom/blob/main/newsloom/newsloom/asgi.py"

        subgraph "Scheduler Tier"
            direction TB
            Scheduler["Stream Scheduler Container\n(run_streams)"]:::scheduler
        end
        click Scheduler "https://github.com/rimamedia/newsloom/blob/main/newsloom/streams/management/commands/run_streams.py"

        subgraph "Integration Services"
            direction TB
            Playwright["Playwright Service"]:::integration
            SlackExt["Slack Integration"]:::integration
            TelegramExt["Telegram Integration"]:::integration
            ArticleanExt["Articlean Service"]:::integration
        end
        click Playwright "https://github.com/rimamedia/newsloom/blob/main/settings/components/playwright.py"
        click SlackExt "https://github.com/rimamedia/newsloom/blob/main/settings/components/slack.py"
        click TelegramExt "https://github.com/rimamedia/newsloom/blob/main/settings/components/telegram.py"
        click ArticleanExt "https://github.com/rimamedia/newsloom/blob/main/settings/components/articlean.py"
    end

    subgraph "Data & Cache"
        direction TB
        Postgres["PostgreSQL"]:::database
        Redis["Redis (Channels/Broker)"]:::cache
    end
    click Postgres "https://github.com/rimamedia/newsloom/blob/main/newsloom/newsloom/settings/components/db.py"
    click Redis "https://github.com/rimamedia/newsloom/blob/main/newsloom/newsloom/settings/components/redis.py"

    Volumes["Static & Media Volumes"]:::storage

    subgraph "External APIs"
        direction TB
        SlackAPI["Slack API"]:::cloud
        TelegramAPI["Telegram API"]:::cloud
        ArticleanAPI["Articlean API"]:::cloud
        ExternalScraping["External Sources"]:::cloud
    end

    Browser -->|HTTP/WS| Nginx
    Nginx -->|HTTP| WebContainer
    Nginx -->|WS| WebContainer
    Nginx -->|Static/Media| Volumes
    WebContainer -->|postgres://| Postgres
    WebContainer -->|redis://| Redis
    WebContainer -->|signals/pubsub| Redis
    Scheduler -->|postgres://| Postgres
    Scheduler -->|calls| Playwright
    Scheduler -->|calls| SlackExt
    Scheduler -->|calls| TelegramExt
    Scheduler -->|calls| ArticleanExt
    Playwright -->|fetch| ExternalScraping
    ExternalScraping -->|data| Scheduler
    SlackExt -->|webhook| SlackAPI
    TelegramExt -->|webhook| TelegramAPI
    ArticleanExt -->|api| ArticleanAPI
    Redis -->|pub/sub| WebContainer

    classDef client fill:#AEDFF7,stroke:#0B6FA4;
    classDef edge fill:#FFD6A5,stroke:#E07A5F;
    classDef web fill:#BDE0FE,stroke:#219EBC;
    classDef scheduler fill:#FFCDB2,stroke:#FF6F61;
    classDef integration fill:#E2F0CB,stroke:#558B2F;
    classDef database fill:#FFCCF9,stroke:#C2185B;
    classDef cache fill:#C8E6C9,stroke:#388E3C;
    classDef storage fill:#E1BEE7,stroke:#7B1FA2;
    classDef cloud fill:#CFD8DC,stroke:#455A64;
```
