How antijob/neuro-parser works
Project Type & Purpose
• Neuroparser is a full-stack application focused on parsing, analyzing, and filtering news and social media content.
• Its main goal is to detect relevant posts or news links based on predefined topics and then notify users via a Telegram bot.
• Additionally, it exposes a REST API for integration and configuration, making it both a backend service and an integration tool.
Analyzing the File Structure
• Top-Level Files: Presence of manage.py and multiple Docker Compose files indicate a containerized Django project with clear separation between development/production environments.
• Server Directory:
• The project is structured following Django’s “apps” approach, with separate submodules like:
○ server/apps/api – Contains REST API views, serializers, permissions, tests, and the URL routes for the external API.
○ server/apps/bot – Implements the Telegram bot, its handlers, command management, fixtures, and related models. This section is responsible for managing Telegram notifications and bot workflows.
○ server/apps/core – Contains business logic, article parsing, index handling, incident prediction, and source parsing. This is the analytical heart of the system and integrates machine learning models.
○ server/apps/users – Manages user-related functions possibly for admin or configuration purposes.
• Celery and Asynchronous Tasks:
• The server/celery directory indicates the system uses Celery for background processing. Typical tasks include crawling, parsing, and ML predictions.
• Core Parsing Modules:
• Directories like server/core/article_parser, server/core/fetcher, server/core/incident_predictor, and server/core/source_parser indicate different “pipelines” for fetching and analyzing content.
• Docker:
Main Elements
a. Components:
• Client Interfaces:
• End Users interacting with the Telegram bot.
• External REST API Clients (or internal admin interfaces) that use the exposed API documentation.
• Web Server (Django):
• REST API endpoints under server/apps/api.
• Admin and static content served via templates.
• Telegram Bot Service:
• Server/apps/bot handling messaging, command processing, and webhook management.
• Background Worker:
• Celery worker processes (from server/celery) handling asynchronous tasks such as news fetching, content parsing, and model inference.
• Core Processing:
• Article Parser & Source Parser components (server/core/article_parser and server/core/source_parser).
• Incident Predictor (server/core/incident_predictor) using BERT or large language model APIs.
• Fetcher component (server/core/fetcher) to retrieve content from external portals.
• Data Storage:
• Underlying database (implied via migrations in various apps) to store parsed data, configurations, and chat IDs.
• External Sources:
• News websites, Telegram channels, social groups (VK, OK).
• External APIs (paid language model APIs).
• Infrastructure Components:
• Docker Compose setup separating the “web”, “bot”, and “celery” containers.
• Load balancers or reverse proxies if applicable (implied in production settings).
b. Relationships and Interactions:
• Data flows from the Fetcher into the Core Processing components (article/source parsers and incident predictors) for analysis.
• Once processed, results are sent to:
• The REST API layer (for administrative interfaces or API consumers).
• The Telegram Bot to dispatch notifications.
• Celery Workers act as intermediaries managing asynchronous operations for these pipelines.
• The Web Server (Django) coordinates requests/responses and handles API calls.
• Docker orchestration coordinates deployment, showing boundaries (e.g., separating containers for “web”, “bot”, and “celery”).
c. Architectural Patterns & Design Principles:
• Separation of concerns: Distinct Django apps for API, Bot, and Core Processing.
• Asynchronous processing: Use of Celery for background jobs.
• Modular design: Independent modules for fetching, parsing, and prediction.
• Containerization: Clean separation of services using Docker.
• Extensibility: Support for different external sources and ML models with configuration.
d. Relevant Technologies and Frameworks:
• Django Framework for the web server and REST API.
• Celery for asynchronous task processing.
• Telegram Bot API for notifications.
• Machine Learning components (BERT models and API integrations for large language models).
• Docker for containerized deployment.
• Testing frameworks (pytest and Django tests directories).
Instructing the Principal Software Engineer
• Create nodes for each major module: Clients (Telegram users, API consumers), Web Server (Django), Telegram Bot Service, Celery Workers, Core Modules (Article Parser, Source Parser, Incident Predictor), Database, External Sources (news sites, social media channels), and Infrastructure (Docker setup).
• From external sources → Fetcher/Parser → Core Processing → Notification (via Telegram Bot)
• REST interactions between clients and the web server.
• Differentiate service layers (e.g., containers/infrastructure vs. business logic vs. integration components).
• Annotations:
• Specify which modules use asynchronous processing.
• Dependencies like the use of machine learning models (BERT, etc.) in the prediction module.
• Configuration boundaries by indicating Docker containers and separate environments (development, production).