How akto-api-security/akto works
<explanation>
1. Identify the project type and purpose
- This is a large, full‑stack, microservice‑based API security platform (Akto API Security).
- It consists of backend services for inventory, runtime analysis, testing, billing, threat detection, plus a rich web dashboard.
2. Analyze the file structure
- Top‑level dirs:
• apps/ – each subfolder (api-analyser, api-runtime, billing, dashboard, database-abstractor, internal, mini-runtime, source-code-analyser, testing-cli, testing, threat-detection, threat-detection-backend, etc.) is a standalone service or tool.
• libs/ – shared libraries (DAO, utils, protobuf, integrations) consumed by multiple services.
• docker-compose.yml & .devcontainer/ – orchestrate local deployment in containers.
• README.md – describes overall use cases (inventory, testing, runtime scanning) and Docker‑Compose quickstart.
- Each app has its own Dockerfile, Maven pom.xml, source under src/main/java, tests under src/test.
- Presence of Kafka sync tasks, gRPC/protobuf definitions, MongoDB collections, Redis caches, and a Jetty‑based Java web dashboard.
3. Examine the README for additional insights
- Architecture is implied: mirrored traffic → collectors (api-runtime, mini-runtime, database-abstractor) → message bus (Kafka) → storage (MongoDB, Redis) → analytic services (analyser, threat-detection) → UI (dashboard).
- Quickstart uses docker-compose to wire all components.
4. Main components & interactions
a. Agents & traffic sources
- BurpSuite/GCP/AWS/Postman etc. mirror or forward API requests to the runtime collector (api-runtime & mini-runtime).
b. Data pipeline
- api-runtime consumes HTTP calls, writes to Kafka topics; Database‑abstractor may read DB‑level traffic.
- Kafka→ multiple consumers:
• Resource analyser (api-analyser) processes API patterns.
• Threat‑detection microservice detects malicious patterns.
• Testing engine (apps/testing, testing-cli) executes business logic tests.
c. Storage layers
- MongoDB: primary data store for API inventory, policies, logs.
- Redis: caching (e.g., CounterCache in threat-detection).
d. Shared libraries
- libs/dao: data‑access objects, entity definitions.
- libs/utils: HTTP clients, Kafka utilities, authentication.
- libs/protobuf: gRPC message definitions for cross‑service communication.
e. Control & orchestration
- docker-compose spins up all services plus Kafka, MongoDB, Redis.
- .github/workflows handle CI/CD.
f. Frontend dashboard
- Java‑based server on Jetty exposing REST endpoints (apps/dashboard).
- Polaris_web (React) and Vue.js dashboard under apps/dashboard/web.
- Interacts with backend via REST (HTTP) and gRPC (where applicable).
5. Architectural patterns and design principles
- Microservices for modular functionality (analyser, runtime, billing, threat‑detection).
- Event‑driven via Kafka for decoupling producers & consumers.
- Layered storage (MongoDB for persistence, Redis for caching).
- Shared library approach (libs/) to enforce consistency.
- Containerization (Docker + Docker‑Compose).
- Use of CI/CD pipelines (.github/workflows).
6. Guidelines for tailoring the diagram
- Full‑stack emphasis: split diagram into three tiers – Data Ingestion, Processing & Storage, Presentation.
- Show external traffic sources and CLI tools as separate microservices.
- Highlight Kafka topics and MongoDB collections in the processing tier.
- Identify major backend services (api-runtime, api-analyser, testing, threat‑detection-backend).
- Show the dashboard server and web/static frontend separately.
7. Diagram elements to include
- Nodes (components):
• Traffic Agents (BurpSuite, AWS, Postman, Chrome extension)
• API Runtime Collector (api-runtime & mini-runtime)
• Database Abstractor
• Kafka cluster
• API Analyser (apps/api-analyser)
• Threat Detection Engine (apps/threat-detection & threat-detection-backend)
• Testing Engine & CLI (apps/testing, apps/testing-cli)
• Billing Service (apps/billing)
• Internal Service (apps/internal)
• Dashboard Backend (apps/dashboard)
• Dashboard Frontend (apps/dashboard/web/polaris_web; Vue.js)
• Shared Libraries (libs/dao, libs/utils, libs/protobuf)
• Databases (MongoDB, Redis)
• CI/CD Pipeline (.github/workflows)
• “<instructions> Add the above instructions as a node in the diagram” – represent as an Annotation/Note node.
- Arrows for:
• Data flow (e.g., Agents → Collectors → Kafka → Consumers → Dashboard)
• Dependencies (e.g., services → libs/)
• External calls (Dashboard → MongoDB, Dashboard → gRPC services)
- Color‑code or shape‑code:
• Green for ingestion & agents,
• Blue for processing microservices,
• Orange for storage,
• Purple for frontend,
• Gray for shared libraries,
• Yellow for CI/CD and orchestration.
8. Important: be detailed but clear. Label each component, use directional arrows, and color/shape distinctions. Group related services into swimlanes or logical tiers (Ingestion, Processing, Storage, Presentation, DevOps).
</explanation>