How 0bytes-security/discovery-ce works
Based on our analysis of the file tree and README, this project is an open-source security assessment automation platform that combines a full‐stack web application with several distinct, integrated components. Its goal is to allow users to integrate various security tools, run automated scans in isolated Docker environments, and get real‐time insights via a modern, type‐safe API and web interface.
Project’s Main Components
• Frontend Client:
• “apps/discovery” directory which contains a Next.js application (using Next.js 14, Shadcn UI, Tailwind CSS, along with authentication pages via Clerk).
• Separate subcomponents such as API routes (e.g., in apps/discovery/src/app/api), authentication (signin/signup subdirectories), dashboard, layout components, and UI elements.
• Backend API:
• “apps/api” directory, which is the FastAPI-based backend.
• Submodules from the “discovery” package (such as core components for configuration, logging, Celery integration, S3 and Pusher utilities, and WebSocket handlers for real‐time updates).
• “discovery/db” folder containing models and repositories (such as registry and runs); this implies data persistence, likely via Tortoise ORM.
• Asynchronous Task Processing:
• Usage of Celery (see apps/api/discovery/core/celery.py) showing how asynchronous tasks are processed (for example, launching and managing security assessment tasks).
• Shared Components & Libraries:
• “libs/shared,” which provides cross-cutting UI components used in the front-end.
• Database / ORM Configuration:
• Integration with Tortoise ORM in the API and also note the presence of “prisma” (e.g., prisma/schema/discovery.prisma) which might indicate database schema definitions or migrations.
• Containerization & Deployment:
• Docker components (found in the “docker/api” and docker-compose files), which are crucial for running assessments in isolated environments.
• Continuous Integration / Continuous Deployment (CI/CD):
• Use of GitHub workflows (located in .github/workflows) like API tests, CodeQL scanning, and deployment of documents.
• Real-Time Communication:
• WebSockets component (apps/api/discovery/ws) that manages real-time feedback from active assessments.
Relationships and Interactions
• Frontend-to-Backend Communication:
• Backend Processing Flow:
• Backend’s API endpoints to:
▪ “core” components for configuration and logging,
▪ WebSocket manager for real-time updates sent back to the frontend.
• Data Persistence Flow:
• How the backend interacts with the database layer (through Tortoise ORM and/or Prisma), including any repositories performing CRUD operations in the “db” folder.
• Container & Deployment Layer:
• That the components (especially the API) are deployed within Docker containers, referencing the Dockerfiles and docker-compose configurations.
Important Architectural Patterns and Design Principles
• Separation of Concerns:
• The project clearly separates front-end (UI, client logic) and backend (API, business logic, task orchestration).
• Modular Design:
• The “discovery” module in the API is divided into containers, core logic, routes, schema definitions, and WebSocket handling – each forming a distinct functional unit.
• Asynchronous Processing:
• Use of Celery demonstrates an event-driven, asynchronous architecture for handling longer-running security assessments.
• Real-Time Feedback:
• The use of WebSockets ensures that clients receive immediate updates in an event-driven fashion.
• CI/CD & Code Quality:
• Strong emphasis on code quality and security, as evidenced by integrated quality tools (CodeQL, Codecov, OpenSSF Scorecard).
Relevant Technologies, Frameworks, and Tools
• Frontend: Next.js 14, React, Tailwind CSS, Shadcn UI, Clerk.
• Backend: FastAPI, Python (with Tortoise ORM for data access), Celery for asynchronous tasks.
• Shared Libraries: Cross-cutting UI components and utilities in “libs/shared.”
• Containerization: Docker and Docker Compose.
• CI/CD: GitHub Actions workflows.
• Testing Tools: Jest (for the discovery app) and various Python testing utilities for API tests.
Elements to Be Included
• Clear, explicit labels for each component (e.g., “Next.js Frontend,” “FastAPI Backend,” “Celery Worker,” “Database/ORM,” “Shared UI Components”).
▪ The client to the backend API,
▪ The API to the asynchronous task processor,
▪ The API to the database,
▪ And the API pushing real-time updates via WebSockets back to the client.
• Visual Differentiation:
Final Advice – Being Detailed
• Bridges between components (e.g., how the frontend communicates with the backend via type-safe APIs, and how tasks flow through Celery).
• Keep in mind to document external dependencies (like Docker environments and CI/CD processes) since they are critical for the operational aspects of this security assessment automation tool.
Happy diagramming!