How rimamedia/newsloom works
Project Type and Purpose
• NewLoom is a full-stack, Django-based news aggregation and publishing platform.
• It combines three pillars:
– A Django web backend (REST API, admin, WebSockets)
– A custom “stream scheduler” for fetching and processing content
– An integration layer (Playwright, Slack, Telegram, Articlean)
File Structure
• Top-level orchestration & infra:
– Dockerfiles (Dockerfile, Dockerfile.ecs, Dockerfile.frontend)
– docker-compose.yml / docker-compose.frontend.yml
– Nginx configs (dev-nginx.conf, nginx.conf, nginx.frontend.conf, nginx_ecs.conf)
– Supervisord configs (supervisord.conf, supervisord.docker.conf)
• Documentation: /docs (Sphinx source)
• Django monorepo structure:
– newsloom/ (project settings, Celery config, ASGI/WGSI entrypoints)
– apps: agents, chat, frontend (API & serializers), mediamanager, sources, streams
– manage.py and custom management commands (e.g., run_streams)
• Static & build scripts: build-frontend.sh, FRONTEND_SETUP.md
• Dependencies & CI/CD: buildspec.yml, GitHub Actions workflows
Extract Key Architecture Notes from README
• Backend: Django 5.1.3, DRF, Channels
• Scheduler: management command run_streams, tracks execution stats
• Integrations: Playwright (JS-heavy scraping), Slack & Telegram bots, Articlean API
• Storage: PostgreSQL (or SQLite), Redis (Channels layer)
• Deployment: Docker + Nginx for static/media, Docker-Compose or ECS
a. Client Side
– Browser SPA or direct REST/WebSocket clients
b. Edge & Load Balancer
– Nginx (HTTP & WS proxy, static file server)
c. Web Tier
– Gunicorn/Uvicorn workers running Django ASGI for HTTP & Channels
d. Scheduler Tier
– A separate container/process executing python manage.py run_streams
– Optionally managed by supervisord or an ECS task
e. Data & Cache
– PostgreSQL cluster
– Redis instance (Channels layer & optional Celery broker)
f. Integration Services
– Playwright headless browsers
– External APIs: Slack, Telegram, Articlean
g. Storage Volumes
– Docker volumes for media and static files
Relationships & Interactions
• Browser → Nginx (HTTP/HTTPS) → Gunicorn/Uvicorn → Django REST endpoints
• Browser → Nginx (WS) → Django Channels → chat_consumer/message events
• Django stores/fetches data from PostgreSQL and uses Redis for pub/sub
• Scheduler process → Streams services → external sources → writes to DB
• On new data, scheduler (or Django signals) publishes updates via Channels to clients
• Django admin UI uses the same web tier & DB
• Playwright runs inside the scheduler container when fetching JS-heavy pages
• Slack/Telegram integrations invoked by stream tasks or chat consumers
Architectural Patterns & Design Principles
• Monolithic Django codebase split into reusable “apps” (agents, chat, sources, streams)
• Layered design: Web/API layer vs. Scheduler layer vs. Integration layer
• Asynchronous messaging: Django Channels + Redis
• Custom pub/sub within Django for real-time notifications
• Management commands as lightweight micro-processes for scheduling
• Docker-first deployment: each component in its own container
Final Touches
• Readability at both high level (overall layout) and detail level (individual flows)