How reworkd/bananalyzer works
Project type and purpose
• Bananalyzer is an open-source Python CLI tool and library for evaluating AI “agents” on web-scraping tasks.
• It packages:
• A command-line interface (__main__.py)
• A core runner engine (bananalyzer/runner/…)
• A data layer for loading example definitions and MHTML snapshots (bananalyzer/data/…)
• A plugin interface (AgentRunner) that users implement to wrap their AI agents
• Optional FastAPI server components (documented but not present in the file tree)
Top-level file structure analysis
• bananalyzer/ – core library
• __main__.py: CLI entrypoint
• runner/: orchestrates test generation & execution
• data/: loads example metadata & snapshots (local JSON, S3, schemas)
• junit.py, hooks.py, schema.py: reporting, event hooks, data models
• static/ – store of MHTML site snapshots & example JSON definitions
• tests/ – pytest test suite verifying each component
• Dev/config files: pyproject.toml, GitHub workflows, scripts
Key README insights
• CLI invokes pytest–based runner which:
a. Loads examples from JSON + MHTML
b. Dynamically builds a test suite
c. Spins up a Playwright BrowserContext
d. Calls AgentRunner.run(context, example) for each example
e. Collects an AgentResult, compares with expected outputs, emits JUnit XML
• Future FastAPI server serves example data via /api/docs.
Main components & interactions (
a. Components
• CLI (bananalyzer command)
• Core Runner Engine
– ExampleLoader (data layer)
– TestGenerator (pytest integration)
– Executor (loops over examples)
• Plugin Interface (AgentRunner implementations)
• Playwright BrowserContext (external service)
• Reporting (JUnit, console)
• Data Stores (static JSON, MHTML, optional S3 buckets)
• (Optional) FastAPI Server
b. Relationships & data flow
1. User → CLI
2. CLI → Runner Engine
3. Runner: load examples via ExampleLoader
4. Runner: create dynamic pytest tests (TestGenerator)
5. For each example:
– Instantiate Playwright context
– Call AgentRunner.run(context, example) → returns AgentResult
– Compare with expected via built-in eval functions
– Emit pass/fail → JUnit reporter
6. Runner aggregates results → console & XML output
7. (If enabled) FastAPI serves same example definitions via HTTP
c. Architectural patterns
• Plugin pattern for user agents (AgentRunner)
• Adapter/wrapper for Playwright API
• Factory for dynamic pytest testcases
• Layered separation: CLI → Core → Plugin → External
d. Technologies & frameworks
• Python 3.6+
• Playwright (async API)
• Pytest for test harness
• FastAPI/Uvicorn (server)
• JSON schemas for validation
• Optional S3 integration for remote snapshots
Detail level
• Sub-modules in Runner: agent_runner.py, generator.py, evals.py, website_responder.py
• Hook points (hooks.py) where custom logic can attach
• How tests map to pytest functions