How jason136/bonfire works
Main Components
• API Server: Represent the HTTP API layer that accepts incoming requests (e.g., POST /prompt_polled, GET /poll_text/{id}, POST /new_streaming, and POST /prompt_streaming/{id}). Indicate that Axum (via Tokio) is the framework handling these endpoints.
• Controller Layer: Illustrate a component that encapsulates request processing logic. The file "src/controller.rs" would be part of this layer.
• Request Extractors & Error Handling: Show supporting components such as extractors ("src/extractors.rs") and error management ("src/error.rs") that process incoming requests and manage invalid or expired responses.
• Text Generation Engine: Represent the core text generation functionality. This includes the distinct modules in "src/text_generation", which house different model implementations. Each file (e.g., "gguf_quantized.rs", "mistral7b.rs", "mixtral.rs", "token_stream.rs", "utils.rs") suggests a separation of concerns for supporting various AI models.
• Polled and Streaming Modules: Distinguish between the text generation strategies. You have a polled text generation endpoint handled by "src/text_polled.rs" and a real-time token streaming endpoint in "src/text_streaming.rs". These modules will orchestrate the asynchronous handling of model outputs (such as streaming or short-term storage for later polling).
Architectural Patterns and Design Principles
• Asynchronous Processing: Note that the project uses Tokio’s async runtime to manage long-running processes (especially important for streaming endpoints).
Specific Technologies
• Rust as the primary language.
• Tokio and Axum for async request handling.
• Huggingface’s Candle for interfacing with generative AI models.
• Any additional dependencies highlighted in Cargo.toml (implied by file structure) can be noted for build and dependency management.