How kientm0810/clone-ctfjudge-for-sure works
CTFJudge combines a new visual workflow MVP with the original scripted evaluation pipeline. In the MVP, users edit and version node graphs, configure judge models, submit golden/student inputs, then inspect asynchronous run status and artifacts; the built-in RAG template chunks and indexes evidence, retrieves relevant chunks, and produces rubric-based comparison output. The graph also preserves the original writeup, trajectory, and qualitative evaluation stages. Evidence is limited to the README and indexed excerpts; unshown API and UI wiring is not asserted.
Main entry points: executor.py Workflow executor · workflows.py Workflow API · llm_service.py LLM service · main.py API application
Components 34
Workflow UI
- Workflow editor
frontend/src/components/workflow/WorkflowEditor.tsx - Run dashboard
frontend/src/components/workflow/WorkflowDashboard.tsx - Model manager
frontend/src/components/workflow/ModelManager.tsx - API client
frontend/src/lib/api.ts
API and persistence
- API application
backend/app/main.py - Workflow API
backend/app/api/workflows.py - Provider API
backend/app/api/providers.py - Deployment API
backend/app/api/deployments.py - Runs API
backend/app/api/runs.py - Workflow store
backend/app/services/workflow_store.py - Workflow and run records
backend/app/models/workflow.py - Provider records
backend/app/models/provider.py
Workflow runtime
- Workflow worker
backend/app/workers/workflow_task.py - Workflow executor
backend/app/workflow/executor.py - Workflow validator
backend/app/workflow/validator.py - Node registry
backend/app/nodes/default_registry.py - Workflow nodes
backend/app/nodes - Execution context
backend/app/workflow/context.py - LLM service
backend/app/services/llm_service.py - Vector service
backend/app/services/vector_service.py - Workflow definitions
backend/app/schemas/workflow.py - Report service
backend/app/services/report_service.py
Original evaluation
- Evaluation runner
run_evaluation.py - Writeup summarizer
writeup_summary_agent.py - Trajectory summarizer
trajectory_summary_agent.py - Qualitative evaluator
qualitative_evaluation_agent.py - LLM response utilities
llm_utils.py - Output versioning
version_manager.py
Other components
- User
- PostgreSQL
- Redis and Celery
- Ollama
- OpenAI-compatible models
- pgvector chunks
Connections 40
- User → Workflow editor: edits workflow
README.md - User → Run dashboard: submits inputs
README.md - User → Model manager: registers models
README.md - Workflow editor → API client: saves definitions
- Run dashboard → API client: runs and polls
- Model manager → API client: manages models
- API application → Workflow API: registers routes
backend/app/main.py - API application → Provider API: registers routes
backend/app/main.py - API application → Deployment API: registers routes
backend/app/main.py - API application → Runs API: registers routes
backend/app/main.py - Workflow API → Workflow validator: validates definitions
backend/app/api/workflows.py - Workflow API → Workflow store: loads and versions
backend/app/api/workflows.py - Workflow API → Workflow executor: executes direct runs
backend/app/api/workflows.py - Workflow API → Vector service: cleans collections
backend/app/api/workflows.py - Workflow store → Workflow and run records: persists workflows
- Workflow and run records → PostgreSQL: stores records
backend/app/models/workflow.py - Provider records → PostgreSQL: stores connections
backend/app/models/provider.py - Provider API → Provider records: manages connections
backend/app/api/providers.py - Workflow worker → Workflow executor: executes queued run
backend/app/workers/workflow_task.py - Workflow worker → Workflow and run records: persists run status
backend/app/workers/workflow_task.py - Workflow worker → LLM service: provides service
backend/app/workers/workflow_task.py - Workflow worker → Vector service: provides service
backend/app/workers/workflow_task.py - Workflow executor → Node registry: loads node handlers
backend/app/workflow/executor.py - Workflow executor → Workflow validator: validates graph
backend/app/workflow/executor.py - Workflow executor → Execution context: builds execution context
backend/app/workflow/executor.py - Node registry → Workflow nodes: registers handlers
- Workflow nodes → Execution context: uses services
backend/app/nodes/database_node.py - LLM service → Ollama: sends chat and embeddings
backend/app/services/llm_service.py - LLM service → OpenAI-compatible models: sends model requests
backend/app/services/llm_service.py - LLM service → Provider records: loads registered model
backend/app/services/llm_service.py - Vector service → LLM service: requests embeddings
backend/app/services/vector_service.py - Vector service → pgvector chunks: indexes and searches
backend/app/services/vector_service.py - Workflow executor → Workflow definitions: consumes definitions
backend/app/workflow/executor.py - Evaluation runner → Writeup summarizer: summarizes writeups
run_evaluation.py - Evaluation runner → Trajectory summarizer: summarizes trajectories
run_evaluation.py - Evaluation runner → Qualitative evaluator: evaluates submissions
run_evaluation.py - Evaluation runner → Output versioning: versions outputs
run_evaluation.py - Writeup summarizer → LLM response utilities: parses responses
writeup_summary_agent.py - Trajectory summarizer → LLM response utilities: parses responses
trajectory_summary_agent.py - Qualitative evaluator → LLM response utilities: parses responses
qualitative_evaluation_agent.py