How ydf0509/funboost works
Project Type and Purpose
• This is a Python library/framework (funboost) for “producer→broker→consumer”-style distributed function scheduling.
• It exposes a single‐decorator API (@boost) that hides a heavyweight feature set (multiple brokers, concurrency models, persistence, monitoring, CLI, web UI).
High-Level Architecture Components
a. User Layer
• User code defines business functions and applies @boost(BoosterParams(...)).
• At runtime, user calls func.push(...) to enqueue and func.consume() to start workers.
b. Core Framework
• Decorator & Params (boost, BoosterParams)
• Configuration loader (funboost_config_getter)
• Task/Message abstractions (message envelope, headers)
• Factories
– PublisherFactory (maps broker_kind → concrete publisher)
– ConsumerFactory (maps broker_kind → concrete consumer)
• Execution Engine
– Concurrency Pool abstraction (concurrent_pool/) supporting threading, asyncio, gevent, eventlet, single_thread + optional multi-process layering
– Task dispatcher that pulls messages, deserializes, acknowledges errors/retries/timeouts, invokes user function
c. Broker Adapters
– Publishers (publishers/) and Consumers (consumers/) modules for each middleware: Redis, RabbitMQ, Kafka, Pulsar, HTTP-queue, disk-queue, SQLAlchemy/Peewee DB queue, Celery/Dramatiq/RQ as broker, etc.
– Helpers in assist/ to embed other frameworks (Celery, Dramatiq, Huey, RQ, FastStream, Rocketry, Taskiq) as brokers.
d. Persistence & Monitoring
• Optional result/status persistence (MongoDB via function_result_status_saver, Peewee/SQLAlchemy via queues/)
• function_result_web/ Flask application (backend + static/js/css + templates) for task result dashboard, speed charts, queue control, RPC calls
• CLI (funboost/core/cli/) powered by fire for command-line queue operations (pause, resume, adjust qps, etc.)
e. Utilities
• Logging (core/loggers, utils/monkey_color_log, nb_log integration)
• Fabric deploy helper for remote task deployment
• Timeout, retry, requeue, rate limiting, distributed QPS control, priority queue, task-filter, scheduling (apscheduler), timing/cron, delay, rpc, kill remote tasks, pause/resume
Interactions & Data Flow
1. User calls some_func.push(args…)
→ Core decorator serializes arguments + metadata → PublisherFactory → concrete publisher → Broker (Redis/RabbitMQ/Kafka/…)
2. Broker enqueues message
3. Consumer process (spawned via some_func.consume())
pulls from Broker → ConsumerFactory → core task dispatcher → Concurrency Pool
4. Concurrency Pool worker
deserializes message → invokes user function → on success/failure executes: ack, retry, requeue, logging, persistence
5. Execution results/status
optionally stored in MongoDB/SQL → polled by Flask web manager → displayed in dashboard
6. CLI & Web Manager
invoke core APIs to pause/resume consumption, clear queue, change QPS/concurrency, view consumer processes
Architectural Patterns & Principles
• Decorator pattern for transparent task registration (@boost)
• Factory/Plugin pattern for broker adapters (publishers/consumers) and concurrency modes
• Adapter pattern for third-party frameworks (Celery, RQ, Dramatiq, Huey)
• Separation of concerns:
• User business logic (functions)
• Message serialization/persistence
• Transport layer (brokers)
• Execution scheduling (concurrency pools)
• Monitoring/UI/CLI