# ydf0509/funboost architecture

> Architecture diagram of the GitHub repository ydf0509/funboost, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ydf0509/funboost
- Repository: https://github.com/ydf0509/funboost

## Overview

##### Project Type and Purpose
- This is a Python library/framework (funboost) for “producer→broker→consumer”-style distributed function scheduling.
- It exposes a single‐decorator API (`@boost`) that hides a heavyweight feature set (multiple brokers, concurrency models, persistence, monitoring, CLI, web UI).

##### High-Level Architecture Components
a. User Layer
- User code defines business functions and applies `@boost(BoosterParams(...))`.
- At runtime, user calls `func.push(...)` to enqueue and `func.consume()` to start workers.

b. Core Framework
- Decorator & Params (`boost`, `BoosterParams`)
- Configuration loader (`funboost_config_getter`)
- Task/Message abstractions (message envelope, headers)
- Factories
– PublisherFactory (maps `broker_kind` → concrete publisher)
– ConsumerFactory (maps `broker_kind` → concrete consumer)
- Execution Engine
– Concurrency Pool abstraction (`concurrent_pool/`) supporting threading, asyncio, gevent, eventlet, single_thread + optional multi-process layering
– Task dispatcher that pulls messages, deserializes, acknowledges errors/retries/timeouts, invokes user function

c. Broker Adapters
– Publishers (`publishers/`) and Consumers (`consumers/`) modules for each middleware: Redis, RabbitMQ, Kafka, Pulsar, HTTP-queue, disk-queue, SQLAlchemy/Peewee DB queue, Celery/Dramatiq/RQ as broker, etc.
– Helpers in `assist/` to embed other frameworks (Celery, Dramatiq, Huey, RQ, FastStream, Rocketry, Taskiq) as brokers.

d. Persistence & Monitoring
- Optional result/status persistence (MongoDB via `function_result_status_saver`, Peewee/SQLAlchemy via `queues/`)
- `function_result_web/` Flask application (backend + static/js/css + templates) for task result dashboard, speed charts, queue control, RPC calls
- CLI (`funboost/core/cli/`) powered by `fire` for command-line queue operations (pause, resume, adjust qps, etc.)

e. Utilities
- Logging (`core/loggers`, `utils/monkey_color_log`, `nb_log` integration)
- Fabric deploy helper for remote task deployment
- Timeout, retry, requeue, rate limiting, distributed QPS control, priority queue, task-filter, scheduling (`apscheduler`), timing/cron, delay, rpc, kill remote tasks, pause/resume

##### Interactions & Data Flow
1. User calls `some_func.push(args…)`
→ Core decorator serializes arguments + metadata → PublisherFactory → concrete publisher → Broker (Redis/RabbitMQ/Kafka/…)
2. Broker enqueues message
3. Consumer process (spawned via `some_func.consume()`)
pulls from Broker → ConsumerFactory → core task dispatcher → Concurrency Pool
4. Concurrency Pool worker
deserializes message → invokes user function → on success/failure executes: ack, retry, requeue, logging, persistence
5. Execution results/status
optionally stored in MongoDB/SQL → polled by Flask web manager → displayed in dashboard
6. CLI & Web Manager
invoke core APIs to pause/resume consumption, clear queue, change QPS/concurrency, view consumer processes

##### Architectural Patterns & Principles
- Decorator pattern for transparent task registration (`@boost`)
- Factory/Plugin pattern for broker adapters (publishers/consumers) and concurrency modes
- Adapter pattern for third-party frameworks (Celery, RQ, Dramatiq, Huey)
- Separation of concerns:
- User business logic (functions)
- Message serialization/persistence
- Transport layer (brokers)
- Execution scheduling (concurrency pools)
- Monitoring/UI/CLI

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User Layer
    subgraph "User Application"
        direction TB
        UserFunc["User Function + @boost"]:::internal
    end

    %% Core Framework
    subgraph "Funboost Core" 
        direction TB
        subgraph "Decorator & Params"
            direction TB
            Booster["booster.py"]:::internal
            ParamsModel["func_params_model.py"]:::internal
        end
        ConfigLoader["funboost_config_getter.py"]:::internal
        subgraph "Serialization & Helpers"
            direction TB
            Serialization["serialization.py"]:::internal
            HelperFuns["helper_funs.py"]:::internal
        end
        subgraph "Factory Layer"
            direction TB
            Map["broker_kind__publsiher_consumer_type_map.py"]:::internal
            PublisherFactory["publisher_factory.py"]:::internal
            ConsumerFactory["consumer_factory.py"]:::internal
        end
        TaskDispatcher["Task Dispatcher"]:::internal
    end

    %% Broker Adapters
    subgraph "Broker Adapters"
        direction TB
        subgraph Publishers
            direction TB
            BasePublisher["base_publisher.py"]:::internal
            RedisPublisher["redis_publisher.py"]:::internal
            RabbitPublisher["rabbitmq_pika_publisher.py"]:::internal
            KafkaPublisher["kafka_publisher.py"]:::internal
        end
        subgraph Consumers
            direction TB
            BaseConsumer["base_consumer.py"]:::internal
            RedisConsumer["redis_consumer.py"]:::internal
            RabbitConsumer["rabbitmq_pika_consumer.py"]:::internal
            KafkaConsumer["kafka_consumer.py"]:::internal
        end
        subgraph "Third-party Frameworks (assist)"
            direction TB
            Assist["assist helpers"]:::internal
        end
    end

    %% Concurrency Engine
    subgraph "Concurrency Engine"
        direction TB
        CPInit["__init__.py"]:::internal
        BasePool["base_pool_type.py"]:::internal
        ThreadPool["custom_threadpool_executor.py"]:::internal
        AsyncPool["async_pool_executor.py"]:::internal
        MultiProc["concurrent_pool_with_multi_process.py"]:::internal
    end

    %% Persistence & Monitoring
    subgraph "Persistence & Monitoring"
        direction TB
        PeeweeQueue["peewee_queue.py"]:::internal
        SqlaQueue["sqla_queue.py"]:::internal
        ResultSaver["function_result_status_saver.py"]:::internal
        ResultConfig["function_result_status_config.py"]:::internal
    end

    %% Web UI & CLI
    subgraph "UI & CLI"
        direction TB
        subgraph "Web Dashboard"
            direction TB
            WebApp["app.py"]:::ui
            WebFunctions["functions.py"]:::ui
            StaticAssets["static/"]:::ui
            Templates["templates/"]:::ui
        end
        subgraph "CLI Interface"
            direction TB
            FunboostFire["funboost_fire.py"]:::ui
            Discovery["discovery_boosters.py"]:::ui
            CliTemplate["funboost_cli_user_templ.py"]:::ui
        end
    end

    %% External Brokers & DBs
    RedisDB[(Redis)]:::external
    RabbitMQ[(RabbitMQ)]:::external
    Kafka[(Kafka)]:::external
    MongoDB[(MongoDB)]:::db
    SQLDB[(SQL DB)]:::db

    %% Flows
    UserFunc -->|"@boost"| Booster
    Booster --> ParamsModel
    Booster --> ConfigLoader
    Booster --> Serialization
    Serialization --> HelperFuns
    Serialization --> Map
    Map --> PublisherFactory
    Map --> ConsumerFactory
    PublisherFactory --> BasePublisher
    PublisherFactory --> RedisPublisher
    PublisherFactory --> RabbitPublisher
    PublisherFactory --> KafkaPublisher
    BasePublisher -->|publish| RedisDB
    BasePublisher -->|publish| RabbitMQ
    BasePublisher -->|publish| Kafka
    ConsumerFactory --> BaseConsumer
    ConsumerFactory --> RedisConsumer
    ConsumerFactory --> RabbitConsumer
    ConsumerFactory --> KafkaConsumer
    RedisDB --> RedisConsumer
    RabbitMQ --> RabbitConsumer
    Kafka --> KafkaConsumer
    RedisConsumer --> TaskDispatcher
    RabbitConsumer --> TaskDispatcher
    KafkaConsumer --> TaskDispatcher
    TaskDispatcher --> CPInit
    TaskDispatcher --> BasePool
    BasePool --> ThreadPool
    BasePool --> AsyncPool
    BasePool --> MultiProc
    ThreadPool -->|"execute"| UserFunc
    AsyncPool -->|"execute"| UserFunc
    MultiProc -->|"execute"| UserFunc
    UserFunc -->|result| ResultSaver
    TaskDispatcher -->|status| ResultSaver
    ResultSaver --> MongoDB
    ResultConfig --> SQLDB
    ResultSaver --> WebApp
    WebApp --> StaticAssets
    WebApp --> Templates

    %% Control Operations
    FunboostFire -.->|control| TaskDispatcher
    Discovery -.->|control| TaskDispatcher
    CliTemplate -.->|control| TaskDispatcher
    WebApp -.->|poll| ResultSaver

    %% Click Events
    click Booster "https://github.com/ydf0509/funboost/blob/master/funboost/core/booster.py"
    click ParamsModel "https://github.com/ydf0509/funboost/blob/master/funboost/core/func_params_model.py"
    click ConfigLoader "https://github.com/ydf0509/funboost/blob/master/funboost/core/funboost_config_getter.py"
    click Serialization "https://github.com/ydf0509/funboost/blob/master/funboost/core/serialization.py"
    click HelperFuns "https://github.com/ydf0509/funboost/blob/master/funboost/core/helper_funs.py"
    click Map "https://github.com/ydf0509/funboost/blob/master/funboost/factories/broker_kind__publsiher_consumer_type_map.py"
    click PublisherFactory "https://github.com/ydf0509/funboost/blob/master/funboost/factories/publisher_factotry.py"
    click ConsumerFactory "https://github.com/ydf0509/funboost/blob/master/funboost/factories/consumer_factory.py"
    click CPInit "https://github.com/ydf0509/funboost/blob/master/funboost/concurrent_pool/__init__.py"
    click BasePool "https://github.com/ydf0509/funboost/blob/master/funboost/concurrent_pool/base_pool_type.py"
    click ThreadPool "https://github.com/ydf0509/funboost/blob/master/funboost/concurrent_pool/custom_threadpool_executor.py"
    click AsyncPool "https://github.com/ydf0509/funboost/blob/master/funboost/concurrent_pool/async_pool_executor.py"
    click MultiProc "https://github.com/ydf0509/funboost/blob/master/funboost/concurrent_pool/concurrent_pool_with_multi_process.py"
    click BasePublisher "https://github.com/ydf0509/funboost/blob/master/funboost/publishers/base_publisher.py"
    click RedisPublisher "https://github.com/ydf0509/funboost/blob/master/funboost/publishers/redis_publisher.py"
    click RabbitPublisher "https://github.com/ydf0509/funboost/blob/master/funboost/publishers/rabbitmq_pika_publisher.py"
    click KafkaPublisher "https://github.com/ydf0509/funboost/blob/master/funboost/publishers/kafka_publisher.py"
    click BaseConsumer "https://github.com/ydf0509/funboost/blob/master/funboost/consumers/base_consumer.py"
    click RedisConsumer "https://github.com/ydf0509/funboost/blob/master/funboost/consumers/redis_consumer.py"
    click RabbitConsumer "https://github.com/ydf0509/funboost/blob/master/funboost/consumers/rabbitmq_pika_consumer.py"
    click KafkaConsumer "https://github.com/ydf0509/funboost/blob/master/funboost/consumers/kafka_consumer.py"
    click Assist "https://github.com/ydf0509/funboost/blob/master/funboost/assist/celery_helper.py"
    click PeeweeQueue "https://github.com/ydf0509/funboost/blob/master/funboost/queues/peewee_queue.py"
    click SqlaQueue "https://github.com/ydf0509/funboost/blob/master/funboost/queues/sqla_queue.py"
    click ResultSaver "https://github.com/ydf0509/funboost/blob/master/funboost/core/function_result_status_saver.py"
    click ResultConfig "https://github.com/ydf0509/funboost/blob/master/funboost/core/function_result_status_config.py"
    click WebApp "https://github.com/ydf0509/funboost/blob/master/funboost/function_result_web/app.py"
    click WebFunctions "https://github.com/ydf0509/funboost/blob/master/funboost/function_result_web/functions.py"
    click StaticAssets "https://github.com/ydf0509/funboost/tree/master/funboost/function_result_web/static/"
    click Templates "https://github.com/ydf0509/funboost/tree/master/funboost/function_result_web/templates/"
    click FunboostFire "https://github.com/ydf0509/funboost/blob/master/funboost/core/cli/funboost_fire.py"
    click Discovery "https://github.com/ydf0509/funboost/blob/master/funboost/core/cli/discovery_boosters.py"
    click CliTemplate "https://github.com/ydf0509/funboost/blob/master/funboost/core/cli/funboost_cli_user_templ.py"

    %% Styles
    classDef internal fill:#bbf,stroke:#333,stroke-width:1px
    classDef external fill:#bfb,stroke:#333,stroke-width:1px
    classDef ui fill:#ffb,stroke:#333,stroke-width:1px
    classDef db fill:#fbb,stroke:#333,stroke-width:1px
```
