How codebyfernanda/desafio_maspacessivel works
MASP Acessível is an evaluation campaign for a MASP guide agent, not the guide-agent runtime itself. Its main workflow runs golden and red-team cases against baseline or corrected Qwen configurations, records answers and RAG evidence, exports AgentCore spans, and evaluates the captured results through two independent fronts: AgentCore evaluators and DeepEval with a local DeepSeek-R1 judge. The repository also preserves comparison outputs and final reporting artifacts. Wiring is based on the README and sampled source; unsampled evaluation modules are represented only where the documented workflow supports them.
Main entry points: campanha.py Campaign runner · avaliacao.py Saved-answer evaluation · executar.py AWS span evaluation · exportar_spans.py Span exporter
Components 21
Campaign inputs
- Golden and attacks
avaliacoes/dados - Baseline and final
avaliacoes/config
Capture and traces
- Campaign runner
avaliacoes/campanha.py - Qwen on AgentCore
- RAG evidence parser
avaliacoes/evidencias_rag.py - Captured responses
avaliacoes/resultados/frente_a - CloudWatch Logs
- Span exporter
avaliacoes/src/agent_core/exportar_spans.py - Exported spans
avaliacoes/resultados/frente_a
AgentCore evaluation
- AWS span evaluation
avaliacoes/src/agent_core/executar.py - Domain rule evaluator
avaliacoes/src/agent_core/lambda_avaliador.py
DeepEval evaluation
- Saved-answer evaluation
avaliacoes/avaliacao.py - Case construction
avaliacoes/src/deepeval/casos.py - Source fidelity metric
avaliacoes/src/deepeval/fidelidade.py - Evaluation prompts
avaliacoes/src/deepeval/templates.py - DeepSeek-R1 judge
- Ollama judge adapter
avaliacoes/src/deepeval/juiz.py
Results and reporting
- Front A results
avaliacoes/resultados/frente_a - Front B results
avaliacoes/resultados/frente_b - Comparison and report
entrega_resumida/RELATORIO.md
Other components
- Evaluator
Connections 27
- Evaluator → Campaign runner: starts run
- Campaign runner → Golden and attacks: loads cases
- Campaign runner → Baseline and final: selects version
- Campaign runner → Qwen on AgentCore: invokes harness
- Qwen on AgentCore → Campaign runner: streams replies
- Campaign runner → RAG evidence parser: analyzes events
- Campaign runner → Captured responses: records answers
- Span exporter → CloudWatch Logs: queries spans
- Span exporter → Captured responses: reads session IDs
- Span exporter → Exported spans: writes spans
- Campaign runner → Span exporter: starts export
- Campaign runner → AWS span evaluation: starts evaluation
- AWS span evaluation → Exported spans: reads spans
- AWS span evaluation → Domain rule evaluator: invokes evaluator
- AWS span evaluation → Front A results: writes scores
- Evaluator → Saved-answer evaluation: starts evaluation
- Saved-answer evaluation → Captured responses: reads captured cases
- Saved-answer evaluation → Case construction: builds test cases
- Saved-answer evaluation → Source fidelity metric: runs fidelity metric
- Saved-answer evaluation → Evaluation prompts: uses prompts
- Saved-answer evaluation → Ollama judge adapter: requests judgments
- Source fidelity metric → Ollama judge adapter: judges claims
- Ollama judge adapter → DeepSeek-R1 judge: sends prompts
- Saved-answer evaluation → Front B results: writes scores
- Front A results → Comparison and report: informs findings
- Front B results → Comparison and report: informs findings
- Comparison and report → Evaluator: presents results