# antijob/neuro-parser architecture

> Architecture diagram of the GitHub repository antijob/neuro-parser, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/antijob/neuro-parser
- Repository: https://github.com/antijob/neuro-parser

## Overview

##### Project Type & Purpose
- Neuroparser is a full-stack application focused on parsing, analyzing, and filtering news and social media content.
- Its main goal is to detect relevant posts or news links based on predefined topics and then notify users via a Telegram bot.
- Additionally, it exposes a REST API for integration and configuration, making it both a backend service and an integration tool.

##### Analyzing the File Structure
- Top-Level Files: Presence of manage.py and multiple Docker Compose files indicate a containerized Django project with clear separation between development/production environments.
- Server Directory:
- The project is structured following Django’s “apps” approach, with separate submodules like:
○ server/apps/api – Contains REST API views, serializers, permissions, tests, and the URL routes for the external API.
○ server/apps/bot – Implements the Telegram bot, its handlers, command management, fixtures, and related models. This section is responsible for managing Telegram notifications and bot workflows.
○ server/apps/core – Contains business logic, article parsing, index handling, incident prediction, and source parsing. This is the analytical heart of the system and integrates machine learning models.
○ server/apps/users – Manages user-related functions possibly for admin or configuration purposes.
- Celery and Asynchronous Tasks:
- The server/celery directory indicates the system uses Celery for background processing. Typical tasks include crawling, parsing, and ML predictions.
- Core Parsing Modules:
- Directories like server/core/article_parser, server/core/fetcher, server/core/incident_predictor, and server/core/source_parser indicate different “pipelines” for fetching and analyzing content.
- Docker:

##### Main Elements
a. Components:
- Client Interfaces:
- End Users interacting with the Telegram bot.
- External REST API Clients (or internal admin interfaces) that use the exposed API documentation.
- Web Server (Django):
- REST API endpoints under server/apps/api.
- Admin and static content served via templates.
- Telegram Bot Service:
- Server/apps/bot handling messaging, command processing, and webhook management.
- Background Worker:
- Celery worker processes (from server/celery) handling asynchronous tasks such as news fetching, content parsing, and model inference.
- Core Processing:
- Article Parser & Source Parser components (server/core/article_parser and server/core/source_parser).
- Incident Predictor (server/core/incident_predictor) using BERT or large language model APIs.
- Fetcher component (server/core/fetcher) to retrieve content from external portals.
- Data Storage:
- Underlying database (implied via migrations in various apps) to store parsed data, configurations, and chat IDs.
- External Sources:
- News websites, Telegram channels, social groups (VK, OK).
- External APIs (paid language model APIs).
- Infrastructure Components:
- Docker Compose setup separating the “web”, “bot”, and “celery” containers.
- Load balancers or reverse proxies if applicable (implied in production settings).

b. Relationships and Interactions:
- Data flows from the Fetcher into the Core Processing components (article/source parsers and incident predictors) for analysis.
- Once processed, results are sent to:
- The REST API layer (for administrative interfaces or API consumers).
- The Telegram Bot to dispatch notifications.
- Celery Workers act as intermediaries managing asynchronous operations for these pipelines.
- The Web Server (Django) coordinates requests/responses and handles API calls.
- Docker orchestration coordinates deployment, showing boundaries (e.g., separating containers for “web”, “bot”, and “celery”).

c. Architectural Patterns & Design Principles:
- Separation of concerns: Distinct Django apps for API, Bot, and Core Processing.
- Asynchronous processing: Use of Celery for background jobs.
- Modular design: Independent modules for fetching, parsing, and prediction.
- Containerization: Clean separation of services using Docker.
- Extensibility: Support for different external sources and ML models with configuration.

d. Relevant Technologies and Frameworks:
- Django Framework for the web server and REST API.
- Celery for asynchronous task processing.
- Telegram Bot API for notifications.
- Machine Learning components (BERT models and API integrations for large language models).
- Docker for containerized deployment.
- Testing frameworks (pytest and Django tests directories).

##### Instructing the Principal Software Engineer
- Create nodes for each major module: Clients (Telegram users, API consumers), Web Server (Django), Telegram Bot Service, Celery Workers, Core Modules (Article Parser, Source Parser, Incident Predictor), Database, External Sources (news sites, social media channels), and Infrastructure (Docker setup).
- From external sources → Fetcher/Parser → Core Processing → Notification (via Telegram Bot)
- REST interactions between clients and the web server.
- Differentiate service layers (e.g., containers/infrastructure vs. business logic vs. integration components).
- Annotations:
- Specify which modules use asynchronous processing.
- Dependencies like the use of machine learning models (BERT, etc.) in the prediction module.
- Configuration boundaries by indicating Docker containers and separate environments (development, production).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Interfaces
    externalSources["\"External Sources\""]:::external
    apiClients["\"API Clients\""]:::client
    telegramUsers["\"Telegram Users\""]:::client

    %% Infrastructure: Docker Orchestration
    dockerInfra["\"Docker Orchestration\""]:::infra

    %% Docker Containers Subgraphs
    subgraph "Web Container"
        webCont["\"Web Container\""]:::infra
        restapi["\"REST API Endpoints\""]:::api
        coreAdmin["\"Core Business Logic\""]:::core
        users["\"User Management\""]:::core
    end

    subgraph "Bot Container"
        botCont["\"Bot Container\""]:::infra
        botService["\"Telegram Bot Service\""]:::bot
    end

    subgraph "Celery Container"
        celeryCont["\"Celery Container\""]:::infra
        celeryWorker["\"Background Worker\""]:::worker
        fetcherComponent["\"Fetcher Component\""]:::worker
        articleParser["\"Article Parser\""]:::worker
        sourceParser["\"Source Parser\""]:::worker
        incidentPredictor["\"Incident Predictor\""]:::worker
    end

    %% Data Storage
    database["\"Database\""]:::infra

    %% Connections for Docker Orchestration to Containers
    dockerInfra -->|"deploys"| webCont
    dockerInfra -->|"deploys"| botCont
    dockerInfra -->|"deploys"| celeryCont

    %% Data Ingestion and Processing
    externalSources -->|"ingest"| fetcherComponent
    celeryWorker -->|"triggers"| fetcherComponent

    fetcherComponent -->|"content"| articleParser
    fetcherComponent -->|"content"| sourceParser
    fetcherComponent -->|"content"| incidentPredictor

    articleParser -->|"results"| restapi
    sourceParser -->|"results"| restapi
    incidentPredictor -->|"results"| restapi

    articleParser -->|"notify"| botService
    sourceParser -->|"notify"| botService
    incidentPredictor -->|"notify"| botService

    %% Client Interactions
    apiClients -->|"request"| restapi
    telegramUsers -->|"message"| botService

    %% Business Data Storage
    coreAdmin -->|"store"| database

    %% Styles
    classDef external fill:#AED6F1,stroke:#1B4F72,stroke-width:2px;
    classDef client fill:#FAD7A0,stroke:#D35400,stroke-width:2px;
    classDef infra fill:#D1F2EB,stroke:#17A589,stroke-width:2px;
    classDef api fill:#ABEBC6,stroke:#229954,stroke-width:2px;
    classDef bot fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef worker fill:#F5B7B1,stroke:#C0392B,stroke-width:2px;
    classDef core fill:#D7BDE2,stroke:#8E44AD,stroke-width:2px;

    %% Click Events
    click restapi "https://github.com/antijob/neuro-parser/tree/main/server/apps/api"
    click botService "https://github.com/antijob/neuro-parser/tree/main/server/apps/bot"
    click celeryWorker "https://github.com/antijob/neuro-parser/tree/main/server/celery"
    click articleParser "https://github.com/antijob/neuro-parser/tree/main/server/core/article_parser"
    click sourceParser "https://github.com/antijob/neuro-parser/tree/main/server/core/source_parser"
    click incidentPredictor "https://github.com/antijob/neuro-parser/tree/main/server/core/incident_predictor"
    click fetcherComponent "https://github.com/antijob/neuro-parser/tree/main/server/core/fetcher"
    click dockerInfra "https://github.com/antijob/neuro-parser/blob/main/docker-compose.yaml"
    click coreAdmin "https://github.com/antijob/neuro-parser/tree/main/server/apps/core"
    click users "https://github.com/antijob/neuro-parser/tree/main/server/apps/users"
```
